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SUMMARY 

The  F-14  Instructional  System  Development  (ISD)  project  had 
two  major  goals.  The  first  goal  was  to  revise  the  existing  F-14 
aircrew  training  system  at  the  Fleet  Readiness  Squadron  (FRS)  level. 
This  effort  was  intended  not  only  to  update  the  content  of  the 
FRS  training  syllabus  but  also  to  ensure  that  a  system  was  designed 
which  was  capable  of  self  maintenance.  The  second  major  goal  of  this 
project  was  to  validate  the  newly  developed  military  specification 
(MIL-I-29053 )  and  its  accompanying  data  item  descriptions  (DID's). 
This  project  was  the  first  large  scale  application  of  the  instruc¬ 
tional  philosophies  and  technology  described  in  the  Military  Speci¬ 
fication  . 

Prior  to  the  start  of  this  project,  a  task  analysis  resulting  in 
a  listing  of  all  the  tasks  performed  by  F-14  pilots  and  Naval  Flight 
Officers  (NFO's)  had  been  completed.  This  task  listing  was  the  basis 
for  the  project  and  the  starting  point  of  the  ISD  process .  From  this 
task  listing,  a  group  of  learning  objectives  was  developed  which 
described  all  the  behaviors,  conditions  and  standards  which  a  student 
aircrew  needed  to  satisfy  in  order  to  achieve  the  desired  training 
results.  The  objectives  were  segregated  and  grouped  into  lessons 
which  comprised  the  events  of  the  training  syllabus. 

One  of  the  most  significant  tasks  performed  as  a  part  of  this 
project  was  the  Training  Support  Requirements  Analysis  (TSRA).  This 
task  analyzed  the  existing  training  system  to  determine  the  strong 
points  and  deficiencies .  The  TSRA  went  on  to  predict  the  future  re¬ 
source  requirements  of  the  FRS's.  Based  upon  this  prediction,  the 
revised  training  system  was  designed  such  that  the  syllabi  were 
tailored  to  those  resources  expected  to  be  available. 

Work  on  this  project  was  performed  primarily  at  VF-124,  NAS 
Miramar,  CA.  A  continuous  interface  was  maintained  with  the  east 
c o a 8 1  FRS,  VF-101  at  NAS  Oceana,  VA.  The  initial  Intent  in  this  pro¬ 
ject  was  to  use  the  Subject  Matter  Experts  (SME's)  at  VF-124  and 
VF-101  to  perform  a  large  portion  of  the  lesson  development  and 
verification  tasks.  This  proved  to  be  an  unachievable  goal  due 
to  the  limited  SME  availability  resulting  from  a  shortage  of  instruc¬ 
tors  (SME's)  at  both  squadrons.  This  problem  of  limited  SME  avail¬ 
ability  has  occurred  consistently  in  other  projects  as  well  as  this 
one  and  can  be  anticipated  to  continue  to  occur  as  long  as  the  Navy 
suffers  from  its  chronic  manpower  shortages. 

This  project  resulted  in  the  development  of  revised  training 
syllabi  tailored  to  the  specific  needs  and  capabilities  of  VF-124 
and  VF-101.  The  training  syllabi  are  now  objective  based  and  provide 
maximum  utility  and  efficiency  in  the  use  of  available  resources. 
Organizational  changes  which  provided  increased  emphasis  on  aircrew 
training  have  been  implemented  within  the  Operations  Departments  at 
the  FRS's.  This  reorganization  also  allows  the  performance  of  the 
quality  control  processes  which  are  essential  in  maintaining  the 
quality  and  efficiency  of  the  training  syllabi.  As  of  the  date  of 
this  report,  the  revised  training  system  is  in  operation  at  both  FRS's 
and  contractor  support  has  been  terminated . 
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PREFACE 


The  Naval  Training  Equipment  Center  has  a  continuing  Interest 
In  the  use  and  evaluation  of  state-of-the-art  procedures  and  me¬ 
thodologies  in  the  design  and  development  of  training  programs. 
Recent  interest  has  centered  on  the  systems  approach  to  training 
now  referred  to  as  Instructional  Systems  Development  (ISD).  The 
effort  discussed  in  this  report  concerns  the  application  of  the 
ISD  process  to  the  development,  implementation,  evaluation  and  re¬ 
vision  of  F-14A  fighter  (MAS)  and  reconnaissance  (TARPS)  aircrew 
training  programs.  The  project,  begun  in  AUG  1977,  was  conducted 
by  the  Naval  Training  Equipment  Center  for  the  Naval  Air  Systems 
Command;  the  work  was  performed  by  Veda,  Inc.  under  contract 
N61339-78-C-0004 . 

The  operational  objectives  of  the  project  were  to  design  an 
aircrew  training  program  that  would  permit  update  as  weapons  system 
hardware  is  modified,  and  to  develop  and  evaluate  the  program 
using  the  ISD  process.  The  research  and  development  objectives 
were  to  gain  experience  in  the  use  of  MIL-T-290S3  and  associated 
DID's  and  to  acquire  resource  utilization  data  for  future  ISD 
planning  and  acquisition. 

Special  appreciation  is  expressed  to  those  personnel  of  VF-124, 
VF— 101 ,  COMNAVAIRPAC,  COMNAVAIRLANT ,  COMFITAEWWINGPAC  and  COMFIT¬ 
WING  ONE  who  made  significant  contributions  to  this  project. 


Robert  G.  Bird 
Acquisition  Director 
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SECTION  I 
INTRODUCTION 


PURPOSE 

The  goal  of  the  F-14  Instructional  System  Development  (ISD) 
-project  was  to  Improve  the  existing  aircrew  training  syllabus  in 
use  at  the  F-14  Fleet  Readiness  Squadrons  (FRS).  As  stated  in  the 
Program  Master  Plan  (PMP),  the  objective  of  the  F-14  ISD  project 
was  to  develop,  produce  and  Implement  a  new  aircrew  training  program 
which : 

a.  is  instructionally  effective  as  measured  by  criterion 
reference  measures  of  student  performance, 

b.  is  efficient  in  terms  of  resource  consumption, 

c.  can  be  adequately  supported  over  its  life  cycle,  and 

d.  continuously  meets  training  objectives. 

The  F-14  training  program  redesign  was  to  be  accomplished  through 
the  application  of  the  ISD  methodologies  specified  in  MIL-T-29053, 
"Training  Requirements  for  Aviation  Weapon  Systems".  MIL-T-29053, 
dated  October,  1977  and  henceforth  referred  to  as  the  military  speci¬ 
fication,  represented  the  most  current  document  designed  to  stan¬ 
dardize  the  processes  and  products  of  aircrew  training  ISD  for  the 
Navy.  This  project  was  the  first  full  scale  training  development 
project  governed  entirely  by  this  specification.  A  second  purpose 
of  this  project  was  therefore  practical  validation  of  the  new  mili¬ 
tary  specification. 

Specifically,  this  project  was  Intended  to  make  the  F-14  syl¬ 
labus  a  more  efficient  and  effective  instrument.  The  F-14  is  one 
of  the  most  complex  and  sophisticated  fighters  ever  procured.  The 
two  man  crew,  the  Pilot  and  the  Naval  Flight  Officer  ( NFO ) ,  are 
responsible  for  the  operation  of  an  aircraft  with  more  air-to-air 
and  air-to-ground  capability  than  has  ever  been  available  before 
in  one  single  airplane.  For  the  first  time  a  true  division  of  re¬ 
sponsibility  and  labor  existed  between  the  two  cockpits. 

This  division  of  responsibility  led  to  the  need  for  partially 
separate  training  syllabi  for  the  pilot  and  the  NFO.  The  number  and 
variety  of  operational  tasks  to  be  mastered  by  the  aircrew  and  the 
high  and  constant  aircrew  replacement  demands  required  syllabi  capa¬ 
ble  of  transmitting  an  immense  amount  of  knowledge  in  a  fairly 
limited  amount  of  time.  To  accomplish  this  demanding  training  task 
a  carefully  and  systematically  designed  training  system  was  needed; 
a  training  system  capable  of  training  each  crew  member  individually 
and  a  crew  jointly  with  an  optimum  of  efficiency. 

The  characteristics  of  the  F-14  also  demanded  a  training  system 
that  could  be  easily  updated.  The  AWG-9  weapon  system,  incorporated 
in  the  F-14 ,  can  be  changed  merely  by  modifying  the  computer  program. 
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This  eooputer  program  can  be  loaded  Into  the  aircraft  In  a  matter 
of  hours.  This  capability,  while  yielding  Immense  benefits  In  terms 
of  cost  and  tactical  adaptability,  creates  a  significant  problem 
In  the  training  system.  A  syllabus  and  a  management  system  which  can 
effectively  Integrate  new  Information  in  a  very  short  time  was 
required . 

SCOPE 

The  F-14  ISD project  Involved  the  analysis  and  revision  of  the 
entire  F-14  training  syllabus.  There  were  no  sections  of  the  syl¬ 
labus  which  were  a  priori  defined  as  unchangeable.  While  this  ap¬ 
proach  undoubtedly  increased  the  total  effort,  it  certainly  con¬ 
tributed  to  the  homogeneity  of  the  end  product.  Integration  of 
existing  materials  with  new  materials  is  always  problematic  due 
to  disparities  in  quality  and  methodology  between  the  existing  and 
new  portions  of  the  syllabus. 


( 


The  scope  of  revising  the  existing  F-14  syllabus  can  best  be 
characterized  by  some  numerical  values.  Overall  length  of  the  syllabus 
prior  to  this  project  was  26  weeks.  The  syllabus  consisted  of  ap¬ 
proximately  250  events  for  the  pilot  and  240  events  for  the  NFO. 

When  expressed  in  terms  of  hours  of  instruction,  the  syllabus  con¬ 
tained  approximately  417  hours  for  Category  I  pilots  and  390  hours 
for  Category  I  NFOs  if  briefings  and  debriefings  are  Included  as  in¬ 
structional  time.  The  precise  tabulation  is  found  In  Table  1.  X. 


Table  1:  Syllabus  Length  (Old  Syllabus) 
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This  syllabus  was  subdivided  into  six  phases  which  were  com¬ 
monly  taught  in  the  following  sequence: 

1.  Transition 

2.  Basic  Fleet  Air  Superiority 

3.  Weapons 

4.  Advanced  Tactics 

5.  Advanced  Fleet  Superiority 

6.  Field  Carrier  Landing  Practice  and  Carrier  Qualification 

Student  throughput  was  200  -  210  students  per  year,  approxi¬ 

mately  equally  divided  between  pilots  and  NFOs .  This  student  load 
was  divided  between  the  two  FRS  squadrons  such  that  VF-124  carried 
approximately  60%  of  the  training  load  and  VF-101  40%. 

The  two  FRS's  have  slightly  dif ferent  syllabi  with  80-90%  simi¬ 
larity.  In  this  project  standardization  of  the  syllabi  between  squa¬ 
drons  was  desired  but  not  absolutely  required.  Priority  was  given 
to  satisfying  the  needs  of  either  squadron,  i.e.,  the  scope  of  the 
project  Included  the  possibility  of  training  program  portions  unique 
to  each  squadron. 

Finally,  the  scope  of  this  project  increased  over  time.  In  1978, 
additional  ISO  needs  were  generated  by  Introduction  of  the  RF-14 
with  its  Tactical  Air  Reconnaissance  Pod  System  (TARPS).  The  training 
requirements  for  the  system  were  extensive  and  eventually  resulted 
in  a  syllabus  addition  of  46  events. 

While  the  numerical  values  above  give  some  indication  of  the 
scope  of  the  project,  the  depth  of  the  work  can  perhaps  best  be 
described  by  the  starting  and  end  point.  The  redesign  of  the  F-14 
course  started  at  the  very  basis,  i.e.,  it  began  with  a  task  analysis. 
This  task  analysis  however,  was  part  of  preceding  efforts  sponsored 
by  COMNAVAIRSYSCOM .  The  present  project  built  on  this  task  analysis 
and  began  with  the  development  of  an  objectives  hierarchy.  The  end 
point  of  the  project  was  Implementation  of  the  course  at  both  squad¬ 
rons.  Implementation  consisted  essentially  of  one  course  administra¬ 
tion  to  one  regular  class  at  either  squadron  and  of  support  in  execu¬ 
ting  the  revisions  indicated  by  these  two  full  scale  tryouts.  Between 
objectives  hierarchy  and  Implementation,  all  processes  specified 
in  the  military  specification  were  executed  and  all  products  de¬ 
scribed  by  the  corresponding  data  item  descriptions  were  delivered. 
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SECTION  II 
BACKGROUND 


REQUIREMENTS 

The  F-14  ISD  program  was  the  second  phase  of  a  continuing  re¬ 
vision  of  the  F-14  FRS  aircrew  training  program.  This  revision  pro¬ 
cess  began  in  1974  with  the  analytic  portions  of  the  ISD  process. 
The  initial  contract  required  completion  of  task  listings,  learning 
objectives  hierarchies,  student  syllabi,  and  lesson  specification 
documents.  All  work  on  this  contract  was  completed  by  January  1977  . 

In  January  1977,  the  Naval  Training  Equipment  Center  (NAVTRA- 
EQUIPCEN)  was  designated  COMNAVAIRSYSCOM  agent  to  execute  the  develop¬ 
ment  of  the  F-14  Aircrew  ISD  program.  In  a  review  of  the  previously 
delivered  documents,  NAVTRAEQUIPCEN  Identified  several  areas  which 
required  additional  effort  to  bring  them  into  conformance  with  the 
newly-developed  Fleet  Aviation  ISD  Model  for  Existing  Weapons  Systems 
at  the  lesson  specification  level  of  development.  Additionally,  NAV¬ 
TRAEQUIPCEN  was  to  conduct  a  problem  analysis  to  determine  program 
goals,  assets  and  constraints. 

The  problem  analysis  was  conducted  and  served  as  a  basis  for 
preparation  of  the  continuation  program.  It  also  acted  as  the  basis 
for  the  development  of  the  F-14  ISD  Program  Master  Plan  (PMP)  which 
was  published  in  August,  1977,  and  revised  in  MAR,  1979.  The  purpose 
of  the  PMP  was  to  coordinate  continuation  of  the  F-14  ISD  program, 
ensuring  that  all  technical,  logistical,  and  managerial  requirements 
for  the  program  had  been  addressed  in  proper  sequence  and  that  roles 
and  responsibilities  had  been  defined. 

Wc  rk  on  the  project  described  in  this  report  began  in  August, 
1977.  As  mentioned  previously,  this  contract  was  governed  by  the 
MIL-T-29053  which  was  published  in  October,  1977.  The  work  performed 
between  the  start  of,  the  contract  and  the  receipt  of  the  Military 
Specification  was  based  on  preliminary  copies  of  this  document  and 
on  continuous  assistance  and  guidance  by  NAVTRAEQUIPCEN. 


ASSUMPTIONS 

INSTRUCTIONAL. 

The  Instructor  workload  could  be  reduced  if  a  greater  proportion  of 
the  syllabus  was  self  instructional .  One  of  the  anticipated  outcomes 
of  this  ISD  project  was  that  the  excessive  workload  of  the  FRS  in¬ 
structors  could  be  reduced.  Prior  to  this  project,  the  primary  mode 
of  instruction  was  the  Interactive  lecture.  Instructor  workload  was 
further  increased  by  the  lack  of  accuracy  or  currentness  of  the 
existing  audio/visual  media .  Most  of  the  self-instructional  events 
required  a  debriefing,  which  amounted  to  another  lecture. 
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The  effectiveness  of  Instruction  In  flight  could  be  Increased  b 
Increasing  the  proportion  of  training  in  the  training  devices.  Th 
ratio  of  Instruction  in  the  trainers  to  that  in  the  aircraft  was 
between  1:2. 5-3  prior  to  this  project.  It  was  assumed  that  the  in¬ 
struction  received  in  the  aircraft  could  be  greatly  improved  by 
Increased  preparation  in  the  training  devices. 

More  relevant  training  could  be  obtained  by  deemphasizing  procedural 
drills  and  by  adding  explicit  Instruction  in  the  area  of  decision 
making  skills.  Prior  to  this  project,  the  training  syllabus  required 
a  great  deal  of  rote  memorization  of  facts  and  procedures.  This  was 
especially  evident  in  the  BOLDFACE  emergency  procedures.  One  of  the 
goals  of  this  project  was  to  add  to  the  instruction  the  rationale 
behind  these  procedures.  It  was  assumed  that  the  correct  performance 
of  procedures  could  be  improved  if  the  student  understood  why  the 
procedures  were  performed. 

The  p 1 1 o 1 8  and  NFO * s  had  very  similar  but  not  identical  training  needs. 
In  the  existing  training  system  the  pilots  and  NFO's  progressed 
together  through  one  syllabus.  One  of  the  primary  goals  of  this 
project  was  the  development  of  a  truly  efficient  training  system 
which  was  also  effective.  This  required  that  the  essential  areas 
of  training  be  emphasized  and  the  non-essential  deemphasized .  In 
that  the  essential  information  differed  for  the  pilot  and  NFO,  se¬ 
parate  but  similar  syllabi  were  developed  for  the  pilot  and  NFO. 

Front-end-loading  could  be  reduced.  The  first  phase,  transition, 
showed  a  very  high  preponderance  of  academic  instruction  and  rela¬ 
tively  little  trainer  and  flight  instruction.  This  is  commonly  re¬ 
ferred  to  as  "front-end-loading  a  syllabus".  A  certain  amount  of 
this  front-end-loading  cannot  be  avoided  when  dealing  with  single 
stick  airplanes  such  as  the  F-14.  Safety  considerations  dictate  that 
the  pilot  acquire  a  great  deal  of  knowledge  on  the  ground  before  he 
can  safely  operate  the  aircraft.  The  same  is  not  true  for  the  NFO, 
therefore,  it  appeared  that  the  pilot  and  NFO  did  not  require  the 
same  degree  of  front-end-loading  in  the  syllabus. 

Sub ject  mat  ter  could  be  restricted  to  what  the  crew  can  either  per¬ 
ceive  or  control,  or  both.  The  existing  training  syllabus  spent 
a  considerable  amount  of  time  teaching  theories  of  operation  and 
detailed  functioning  of  mechanical  systems.  In  a  program  involving 
a  system  as  complex  as  the  F-14,  efficiency  cannot  be  achieved  un¬ 
less  the  focus  of  training  is  oriented  to  crew,  i.e.,  how  do  I  operate 
it?,  vice  how  does  it  function? 

The  instructional  resources  which  existed  within  VF-124  and  VF-101 
were  to  be  utilized  as  much  as  possible  during  the  development  of 
the  revised  syllabi.  It  was  hoped  that  many  of  the  instructional 
events,  particularly  the  more  expensive  slide/tape  presentations, 
could  be  used  in  the  new  syllabi.  However,  it  was  clearly  delineated 
that  no  event  or  phase  of  the  syllabus  was  considered  "sacred". 
The  ISD  methodology  was  to  be  prosecuted  and  existing  resources  were 
not  to  be  a  hindrance  in  this  effort. 
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The  need  for  a  continuous  output  of  graduates  from  the  FRS's  would 
continue  throughout  this  contract  including  the  period  of  Implemen¬ 
tation  .  It  was  not  deemed  feasible  to  shut  off  the  input  of  students 
or  the  output  of  graduates.  The  needs  of  the  operational  squadrons 
could  not  be  altered  due  to  the  syllabus  revision  effort.  An  imple¬ 
mentation  plan  was,  therefore,  required  to  ensure  that  the  new  syl¬ 
labus  could  be  placed  in  operation  without  disrupting  the  flow  of 
graduates  . 

The  training  goals  could  be  achieved  with  a  syllabus  not  exceeding 
26  weeks  in  length.  Based  on  experience  with  the  old  syllabus  the 
FRS  training  goals  could  be  reached  within  26  weeks  given  good  re¬ 
source  availability  and  fair  weather.  This  26  week  limit  was  not  to 
be  exceeded  by  the  new  syllabus.  If  application  of  the  ISD  process 
could  lead  to  a  syllabus  of  less  than  26  weeks  in  length,  no  objec¬ 
tions  would  be  raised.  However,  due  to  numerous  factors  such  as  the 
availability  of  students  and  the  need  for  regular  replacements  in 
the  fleet,  a  limit  of  26  weeks  was  set.  This  was  the  length  of  the 
syllabus  which  existed  prior  to  this  revision  effort. 


TECHNOLOGICAL. 

The  ISD  technology  described  in  the  military  specification  and  asso¬ 
ciated  DID's  was  sufficient,  but  in  need  of  validation.  It  was  as¬ 
sumed  at  the  time  that  the  military  specification  and  the  DID's 
were  sufficient  in  detail  to  perform  all  the  tasks  requested.  The 
content  of  the  military  specification  was  well  known  to  the  instruc¬ 
tional  designers  working  on  this  project.  The  ISD  technology  was 
up  to  date  in  every  way.  Still,  this  set  of  DID's  and  specification 
had  not  been  validated  in  the  actual  performance  of  a  full  scale 
ISD  project.  The  F-14  ISD  project  was  one  of  several  to  provide 
that  validation. 

The  existing  task  analysis  was  useable  without  rework.  This  task 
analysis  had  been  done  as  part  of  a  previous  effort.  An  appraisal 
of  this  analysis  was  made  and  it  was  determined  that  it  could  be 
used  without  modification.  The  development  of  the  learning  objectives 
hierarchies  was  therefore  the  first  task  in  this  project. 

The  physical  resources  for  performing  training  at  the  FRS's  were 
sufficient.  During  the  early  stages  of development  of  VF-124,  deci¬ 
sions  were  made  and  implemented  which  resulted  in  the  creation  of 
a  modern  academic  center.  Classrooms  and  study  carrels  with  modern 
audio  visual  equipment  were  procured  and  located  in  the  building 
housing  the  F-14  training  devices .  This  building  was  air-conditioned 
and  away  from  the  noise  of  the  flight  line.  It  provided  a  quiet, 
comfortable  atmosphere  conducive  to  study  and  learning.  As  a  result 
of  the  success  of  the  academic  facility  at  VF-124,  similar  equipment 
was  procured  for  VF-101. 

The  existing  syllabus  was  supported  by  separate  front  and  back  seat 
simulators  which  could  be  linked  for  joint  operation.  These  simulators 
were  considered  sdequate  for  the  new  syllabus. 
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The  Air  Combat  Maneuvering  Range  (ACMR)  debriefing  facility  could 
provide  valuable  pretraining  If  used  in  a  pro-active  mode.  The  ACMR 
is  a  facility  in  which  aircraf  t  par  t  icl  pat  1  ng  In  simulated  air  combat 
are  tracked  via  telemetry.  This  allows  the  simulated  combat  to  be 
viewed  from  a  ground  facility  on  computerized  display  screens.  The 
telemetry  is  also  stored  on  tape  for  later  viewing  by  the  aircrews 
who  engaged  in  the  simulated  combat.  It  was  assumed  that  these 
tapes  could  provide  training  if  used  in  a  pro-active  mode.  In  this 
scheme,  students  would  observe  selected  tapes  of  previous  engagements. 
At  certain  points  the  tape  could  be  stopped  and  the  students  would 
be  questioned  as  to  the  probable  results  of  the  various  options 
available  to  the  aircrews  at  that  point.  The  tape  could  be  restarted 
and  the  results  of  one  possible  decision  could  be  examined.  This 
would  allow  the  crews  to  Improve  their  decision  making  abilities  in  an 
Inexpensive  and  efficient  manner,  not  Involving  flight  time  expense. 


MANAGEMENT. 

The  contractor  was  responsible  for  the  design  of  the  product.  The 
DID  * s  and  ml 11 ta  ry  specification  were  to  be  u  sed  as  guidance  in  the 
performance  of  all  tasks.  However,  it  was  expected  that  some  Innovation 
would  result  from  the  performance  of  the  project.  Besides  validating 
the  DID's  and  military  specification,  it  was  hoped  that  the  state- 
of-the-art  in  aircrew  training  would  be  advanced. 

Navy  SME ' s  from  VF-124  and  VF-101,  would  support  the  project.  The 
development  of  the  training  program  was  to  be  supported  by  Navy 
SME ' s  with  80  hours  per  week  as  stated  in  the  Program  Master  Plan. 
This  support  was  to  be  coordinated  by  a  full  time  SME  team  leader, 
the  ISD  officer.  The  tasks  to  be  performed  by  the  Navy  SME  '  s  consisted 
of  lesson  specification  development,  review  of  lesson  specifications 
developed  by  contractor  SME's,  and  in  formative  evaluation  of  the 
training  materials  produced  from  those  lesson  specifications. 

The  overall  coordination  of  the  project  was  to  be  accomplished  by  a 
series  of  Fleet  Steering  Committee.  The  Fleet  Steering  Committee, 
chaired  by  VF-124,  consisted  of  representatives  from  the  type  and 
wing  commanders,  and  from  VF-124  and  VF-101  ISD  departments.  Personnel 
from  NAVTRAEQUIPCEN  and  the  two  contractors  Involved  in  this  project 
also  attended  the  meetings. 

The  Automated  Training  Support  System  (ATSS),  a  computer  assisted 
managment  tool,  would  be  available  during  this  project.  The  primary 
functions  anticipated  to  be  performed  by  ATSS  were  scheduling  and 
record  keeping.  The  design  of  ATSS  was  to  allow  the  input  of  the 
syllabus  and  the  available  resources,  and  provide  an  output  of  the 
dally  academic , trainer  and  flight  schedules.  These  schedules  would 
be  derived  from  the  accomplishments  of  the  previous  schedules.  Record 
keeping  of  individual  student  progress  and  achievement  was  a  clerical 
function  necessary  to  derive  the  new  schedule. 
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CONSTRAINTS 

Separate  training  syllabi,  training  philosophies  and  management  or¬ 
ganizations  existed  at  VF-124  and  VF-101.  The  syllabi  which  were 
developed  differed  for  the  two  squadrons.  Likewise,  the  management 
organizations  and  therefore  the  procedures  for  management  differed 
■slightly.  The  needs  of  both  organizations  had  to  be  satisfied  by  the 
ISD  effort . 

Purther,  both  the  development  and  implementation  of  the  revised 
syllabi  had  to  be  done  on  a  non-interfering  basis.  The  operations 
of  the  squadrons  could  not  be  suspended.  The  need  for  graduates  on 
a  regular  basis  would  remain  constant  throughout  the  project. 

There  would  be  a  maximum  time  limit  of  a  26  weeks  for  the  syllabus. 
This  time  limit  was  established  by  CNO  based  upon  pragmatic  and 
historical  factors.  These  included  the  need  to  input  and  graduate 
students  based  on  their  availability  from  the  training  command  and 
the  demands  of  the  fleet.  Historically,  other  fighter  training  syllabi 
(F-4  in  particular)  were  accomplished  in  that  same  time  period. 

Major  modifications  to  the  training  devices  were  not  to  be  proposed 
as  a  part  of  this  ISD  effort.  The  result  of  this  situation  was  that 
the  syllabi  had  to  be  designed  around  the  capabilities  and  limita¬ 
tions  of  these  devices. 

Student  progress  through  the  syllabuB  was  to  be  in  groups  (classes) 
vice  individually.  Navy  management  desired  to  maintain  the  class 
structure  which  has  historically  existed  in  the  FRS  '  s  .  The  contractor 
had  proposed  that  the  syllabus  be  designed  for  individual  progress. 
However,  the  Navy  preferred  the  retention  of  the  class  system  because 
of  the  ease  of  management  of  small  groups  vice  individuals,  and 
the  benefits  of  cooperation  and  competition  among  the  class  and 
between  classes. 
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SECTION  III 


IMPLEMENTATION 


ORGANIZATION 

Figure  1  describes  Che  commands  and  organizations  Involved  with 
this  project.  The  ultimate  authority  and  responsibility  for  this 
project  rested  with  the  Chief  of  Naval  Operations  (CNO).  The  Fighter 
Training  Branch  (4132)  of  the  Naval  Air  Systems  Command  (COMNAVAIR- 
SYSCOM)  acted  as  the  procuring  agency,  i.e.,  the  source  of  the 
funds.  NAVTRAEQUIPCEN  was  designated  as  COMNAVAIRSYSCOM's  agent 
to  contract  for  and  administer  the  project.  On  the  military  side, 
the  chain  of  command  extends  from  CNO  to  the  type  commanders  (COM- 
NAVAIRPAC  and  COMNAVAIRLANT) ,  to  the  cognizant  wing  commanders  (COM- 
FITAEHWINGPAC  and  COMFITWINGONE)  to  the  Fleet  Readiness  Squadrons 
(VF-124  and  VF-101).  The  primary,  or  most  frequently  used,  channel 
of  communication  was  between  NAVTRAEQUIPCEN  and  the  two  ISD  officers 
at  the  FRS's. 

In  the  implementation  of  this  project,  two  separate  contractors 
performed  separate  but  related  tasks.  Veda,  Inc.  was  contracted 
to  perform  the  ISO  project  described  herein,  and  Perspective  Instruc¬ 
tional  Communications,  Inc.,  a  media  production  firm,  was  contracted 
to  produce  the  audio-visual  and  printed  materials  based  on  lesson 
specifications  developed  by  Veda.  Some  initial  problems  were  encoun¬ 
tered  in  this  arrangement  due  to  the  lack  of  a  contractual  agreement 
specifying  the  responsibilities  of  the  two  contractors  to  each  other. 
This  problem  was  solved  with  a  letter  of  agreement  between  the 
two  contractors. 

Figure  2  describes  the  relationship  which  existed  during  the 
performance  of  this  contract.  The  contractor's  General  Manager  and 
Technical  Director  were  responsible  to  NAVTRAEQUIPCEN  for  the  per¬ 
formance  of  all  tasks.  Their  Jobs  included  the  direction  and  super¬ 
visions  of  the  project  manager  and  the  generation  of  all  reports 
and  documentation  required. 

The  project  manager  interfaced  primarily  with  the  ISD  officers 
at  VF-124  and  VF-101.  The  project  manager  was  responsible  for  the 
coordination  of  the  contractors  staff.  In  conjunction  with  the  ISD 
officer,  the  project  manager  coordinated  the  efforts  of  his  staff 
and  the  SME's  supplied  by  the  ISD  officer.  All  work  performed  by 
the  contractor  was  originated  or  performed  by  the  project  manager 
under  the  guidance  and  direction  of  his  technical  director  end  general 
sianager . 

This  organizational  relationship  was  found  to  be  very  effective. 
The  working  level  personnel,  the  contractors  staff  and  the  Navy 
SME's  interfaced  with  each  other  in  generating  the  revised  syllabus. 
The  administrative  and  contractual  interfaces  were  separated  from 
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this  working  environment  and  performed  by  the  appropriate  managers 
and  contracting  officers.  It  should  be  noted  that  at  the  beginning 
of  this  project  the  Technical  Director  and  the  Project  Manager  were, 
in  fact,  the  same  person.  As  the  scope  of  the  project,  and  thus  the 
staff,  increased  a  separate  project  manager  was  appointed. 

The  final  organizational  structure  which  should  be  discussed 
is  that  of  the  FRS's.  This  discussion  will  trace  the  changes  which 
occurred  in  the  organization  of  VF-124,  the  points  of  the  discussion 
also  pertain  to  VF-101. 

At  the  beginning  of  this  project  the  organization  of  VF-124  was 
almost  identical  to  that  of  an  operational  fighter  squadron.  An 
operations  officer,  as  a  department  head,  was  responsible  for  all 
flight  operations  and  for  all  training.  The  maintenance  officer, 
however,  was  responsible  only  for  the  maintenance  of  the  squadron's 
aircraft.  Training  for  fleet  replacement  maintenance  personnel  was 
performed  by  the  Fleet  Replacement  Aviations  Maintenance  Personnel 
(FRAMP)  department.  The  FRAMP  officer  was  a  department  head  with 
status  equal  to  the  Operations  and  Maintenance  officers. 

The  contractor  recommended  that  an  organizational  structure 
similar  to  the  FRAMP  structure  be  considered  for  aircrew  training. 
Most  significantly,  the  reorganization  was  to  Include  an  Aircrew 
Training  Department  on  the  same  level  as  the  Operations  Department. 

During  the  course  of  this  project,  a  division  of  responsibility 
between  the  Training  Department  and  the  Operations  Department  was 
realized.  The  Jobs  of  operating  the  squadrons  aircraft  and  the 
overall  management  of  the  aircrews  were  separated  from  the  jobs 
of  developing  and  performing  aircrew  training.  The  organizational 
structure  now  in  use  at  VF-124  is  shown  in  Figure  3.  It  was  decided 
by  VF-124  that  an  Operations  Officer  would  retain  control  over  aircrew 
training,  however,  two  sub-departments  called  TRAINING  MANAGEMENT 
and  TRAINING  DEVELOPMENT/MODEL  MANAGER,  would  functionally  effect 
the  operation  of  the  department.  The  emphasis  on  the  development 
and  performance  of  training  was  achieved  by  this  organizational 
structure  . 
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STAFFING 

The  Veda  staff  was  composed  of  an  average  of  9  persons  during 
the  course  of  this  project.  The  staff  Included  two  PHD's  who  acted 
as  instructional  psychologists  and  provided  the  primary  design  and 
management  of  the  instructional  products.  Assisting  were  4  instruc¬ 
tional  designers  with  completed  masters  degrees  in  educational  tech¬ 
nology  or  psychology.  The  remainder  of  the  staff  acted  as  SME's. 
The  staff  expertise  and  experience  varied.  Most  were  former  instruc¬ 
tors  from  the  F-14  FRS ' s .  Their  educational  backgrounds  ranged  from 
bachelor  to  master  degrees. 

The  combination  of  instructional  design  expertise  with  contrac¬ 
tor  SME's  proved  to  be  an  ideal  staffing  method.  The  SME's,  working 
full  time  with  the  Instructional  designers,  gained  valuable  expertise 
in  the  field  of  ISD.  The  Instructional  designers  were  likewise  able 
to  gain  knowledge  in  the  fields  of  aviation  and  Navy  aircrew  training. 
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PROJECT  TASKS 


SUBJECT  MATTER  EXPERT  (SME)  TRAINING  MATERIALS. 


Duration  of  Task:  November,  1977  -  March,  1978 
Mandays  Expended: 


Program  Manager  8 
Instructional  Psychologist  I  11 
Instructional  Psychologist  II  59 
Instructional  Technologist  I  0 
Subject  Matter  Expert  5 
Clerical  23^ 

TOTAL . 106 


Subject  Matter  Expert  (SME)  is  a  term  which  is  generally  ac¬ 
cepted  in  the  instructional  design  field  as  pertaining  to  those 
individuals  who  have  extensive,  first-hand  knowledge  of  technical 
content.  In  the  specific  context  of  the  F-14  training  program,  the 
subject  matter  experts  were  Navy  instructor  F-14  pilots  and  naval 
flight  officers  (NFO)  f rom  VF-124  and  VF-101  who  had  current  and  in¬ 
timate  knowledge  of  the  weapon  system. 

Navy  SMEs  were  utilized  during  the  F-14  training  program  design 
phase  to  provide  information  to  the  contractor's  Instructional  de¬ 
signers  and  to  ensure  technical  accuracy  of  the  lesson  materials. 
SME  involvement  was  necessary  in  providing  input  to  the  lesson  speci¬ 
fication  portion  of  the  design  phase  and  to  the  formative  evaluation 
of  lesson  materials  during  development.  Contractual  arrangements 
were  made  for  80  hours  of  SME  participation  per  week.  Course  materials 
were  developed  to  provide  the  SMEs  with  a  general,  overall  under¬ 
standing  of  the  training  program  design  process,  and,  more  speci¬ 
fically,  the  lesson  specification  and  formative  evaluation  require¬ 
ments.  It  was  not  desired  that  these  subject  matter  experts  become 
experienced  Instructional  designers,  but  that  they  be  capable  of 
understanding  the  instructional  systems  development  (ISD)  process 
and  their  role  in  developing  lesson  specifications  and  in  formative 
evaluation. 

The  process  of  developing  the  SME  training  materials  consisted 
of  six  steps:  (See  Figure  4) 


Figure  4 
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STEP  I 


SME  Training  Materials  Development 
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STEP  1.  Analyze  needs  for  SME  course  materials.  The  needs  for  the  SHE 
course  materials  were  based  upon  the  three  requirements  specified 
in  the  DID.  Those  requirements  were  that  the  SME '  s  (1)  have  a 
overall  understanding  of  the  ISD  process,  (2)  provide  generality 
support,  instance  specifications  and  practice  and  test  items,  and 
(3)  evaluate  the  course  materials,  aid  in  conducting  small  group 
tryouts  and  aid  in  data  collection. 

STEP  2.  List  the  tasks  required  to  satisfy  the  needs  determined  in 


tep  1 .  Contractor  instructional  technologists,  surveyed  the  pro¬ 
cesses  Involved  in  accomplishing  the  requirements  stated  in  Step 

1.  The  end  products  or  processes  to  be  produced  or  performed  were 
listed  and  analyzed  to  determine  individual  tasks.  The  tasks  were 
then  listed  and  associated  with  their  corresponding  requirements 
or  goal. 

Of  the  three  requirements  determined  in  Step  1,  only  two  of  the 
three  had  tasks  associated  with  them.  The  tasks  corresponding  to 
the  three  goals  are  listed  below. 

Goal  1:  Understand  the  meaning  of  the  ISD  process  and  its  use  for 
the  F-14  training  program. 

Tasks  corresponding  to  Goal  Is 


No  tasks. 

Goal  2:  Give  appropriate  sub ject matter  input  to  the  contractor 
provided  lesson  specifications. 

Tasks  corresponding  to  Goal  2: 


1.  Review  contractor-supplied  generalities. 

2.  Write  support  generalities  for  the  generalities  provided 
in  the  lesson  specifications. 

3.  Write  examples  and  non-examples  for  concept  and  rule-using 
generalities  . 

4.  Provide  common  error  analysis  for  concept  and  rule-using 
generalities  . 

5.  Help  contractor  instructional  designers  in  writing  test 
items  for  each  objective  within  each  lesson. 

Goal  3:  Understand  the  meaning  of  formative  evaluation  and  be 
able  to  give  appropriate  feedback  in  the  one-to-one 
and  small  group  tryouts. 

Tasks  corresponding  to  Goal  3: 


!•  Review  lessons  for  content  and  technical  accuracy. 
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2.  Aid  In  the  conduct  of  small  group  tryouts. 

3.  Serve  as  a  subject  in  Individual  tryouts. 

4.  Administer  tests. 

5.  Score  tests. 


STEP  3.  Develop  learning  objectives  based  upon  the  tasks  listed  in 
Step  2.  The  SME  objectives  were  derived  from  the  task  listing  by 
asking  the  question,  "What  must  the  SME  know  in  order  to  perform 
this  task?”.  In  the  case  of  Goal  1,  where  there  are  no  tasks,  the 
procedure  for  defining  the  objectives  directly  from  the  goal  is 
termed  "operationalizing".  This  procedure  is  essentially  the  same 
as  when  deriving  the  objectives  from  tasks  and  asks  the  question, 
"What  must  the  SME  know  in  order  to  achieve  this  goal?”.  The  fol- 
lowing  are  examples  of  objectives  which  were  determined  in  thi6 
manner : 

A.  The  ISD  Process 
(1)  Objectives 

The  SME  will  be  able  to  list  four  major  advantages  of  the 
ISD  process  of  instructional  design. 

Given  a  list  of  ISD  components  and  a  list  of  possible  de¬ 
finitions,  the  SME  will  be  able  to  match  all  the  components 
with  their  appropriate  definition. 

B.  Lesson  Specifications 

(1)  Objectives  (examples) 

Given  an  objective  and  corresponding  generality,  the  SME 
will  be  able  to  recognize  the  appropriate  generality  support. 

Given  a  concept  or  rule-using  objective  and  generality,  the 
SME  will  be  able  to  identify  the  types  of  examples  and 
non-examples . 

C.  Formative  Evaluation 

(1)  Objectives  (examples) 

Given  various  alternatives,  the  SME  will  be  able  to  recog¬ 
nize  the  types  of  information  desired  from  an  individual 
t  ryout . 

Presented  with  lesson  materials,  the  SME  will  be  able  to  eval¬ 
uate  the  materials  for  technical  accuracy  and  completeness . 
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STEP  4;  Select  media.  The  selection  of  media  for  the  SME  course 
materials  was  based  upon  the  following  considerations:  Who  are  the 
learners?  What  type  of  information  is  to  be  delivered?  What  method 
of  delivery  is  most  cost  effective? 

The  learners  were  Navy  instructors  from  VF-124  and  VF-101  who 
worked  long  and  varying  hours.  Since  there  was  little  time  and  few 
opportunities  to  conduct  group  training,  and  since  there  was  a  con¬ 
tinuous  turnover  in  the  instructor  corps,  it  was  deemed  necessary 
that  the  instruction  be  readily  available  at  any  time.  Also,  some 
of  the  SME's  were  located  at  VF-101  where  there  was  little  availa¬ 
bility  for  interface  with  the  contractor.  The  materials,  therefore, 
had  to  be  easily  transf errable . 

The  types  of  information  presented  in  the  SME  course  materials 
were  for  the  most  part,  facts,  concepts  and  rules.  This  type  of 
learning  can  be  done  without  an  instructor  and  with  little  or  no 
visual  display  requirements. 

With  this  in  mind,  the  most  cost  effective  method  of  mediation 
for  SME  course  materials  was  clearly  a  print  ins  trument ,  modularized, 
and  packaged  for  individual  use.  The  lessons  for  the  SME  course 
materials  were  identified  and  sequenced  especially  to  be  presented 
by  print  materials  in  an  individualized  package  where  the  SME  can 
work  through  as  many  or  as  few  of  the  lessons  or  modules  as  he  desires 
or  has  time  for  in  a  single  sitting. 


STEP  5:  Develop  the  syllabus/materlals .  Syllabus  development  was 

accomplished  by  first  grouping  objectives  into  lessons.  The  rationale 
for  grouping  objectives  was  essentially  one  of  conceptual  relation¬ 
ships.  In  addition,  certain  learning  principles  were  applied  to  the 
conceptual  relationships  of  objectives  in  order  to  ensure  their  in¬ 
structional  compatibility.  During  this  process  the  following  ques¬ 
tions  were  asked: 

a.  Does  the  set  of  objectives  for  this  lesson  represent  a  co¬ 
herent  unit? 

b.  Are  all  prerequisite  objectives  to  the  lesson  noted? 

c.  Are  there  too  many  or  too  few  objectives  in  the  lesson? 

d.  Is  the  lesson  too  long  or  too  short? 

When  the  lessons  were  identified,  they  were  then  sequenced  into 
the  optimal  order  for  instruction .  This  ordering  was  based  on  prere¬ 
quisite  information  and  logical  relationships .  More  precisely,  in 
the  case  of  ordering  the  lessons  for  subject  matter  experts,  the 
order  in  which  each  of  the  components  occurs  in  the  lesson  specifi¬ 
cation  provides  some  logic  for  sequencing.  Even  though  the  learner 
does  not  necessarily  have  to  learn  how  to  write  a  generality  support 
before  he  learns  how  to  develop  an  instance  specifications,  it  seemed 
logical  to  have  him  learn  the  material  in  this  sequence  because 
one  occurs  before  the  other  in  the  lesson  specification. 
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STEP  6:  Conduct  formative  evaluation  of  the  SME  training  course  ma¬ 
terials.  A  formative  review  of  SME  course  materials  was  conducted 
prior  to  final  development.  Completed  lesson  specifications  were 
given  to  Navy  SMEs  for  evaluation.  Based  on  their  understanding  and 
feedback  from  these  materials,  the  SME  course  was  revised  and  produced. 

The  Subject  Matter  Expert  training  materials  development  task 
was  one  of  the  first  performed  as  a  part  of  this  project.  This  was 
required  because  of  the  need  for  trained  SME's  during  the  lesson 
specification  development  phase.  The  SME's  would  later  aid  in  the 
formative  evaluation  of  the  training  materials.  Timeliness  was  not 
an  issue  here. 

The  early  development  of  SME  training  materials  was  essential 
for  another  reason  unrelated  to  the  performance  of  the  various  tasks. 
The  first  portion  of  the  SME  training  materials  that  deals  with  the 
1SD  process  proved  to  be  instrumental  in  “selling*'  the  ISD  philosophy 
to  the  Instructor  corps.  If  the  enhancement  of  the  existing  training 
system  was  to  be  successful,  it  was  essential  that  the  support  of  the 
entire  Instructor  corps  be  gained.  A  clear  presentation  of  the  goals 
and  processes  involved  in  the  project  proved  to  be  successful  in 
gaining  that  support. 

An  example  of  the  SME  training  materials  produced  during  this 
task  can  be  found  in  Appendix  A. 


OBJECTIVES  HIERARCHIES. 

Duration  of  Task:  August,  1977  -  May,  1980 
Mondays  Expended: 


Program  Manager  76 
Instructional  Psychologist  I  76 
Instructional  Psychologist  II  137 
Instructional  Technologist  I  46 
Subject  Matter  Expert  59 
Clerical  _8 

TOTAL . 402 


This  section  details  two  major  processes  concerning  the  learning 
objectives  hierarchies.  The  first  process,  which  occurred  at  the 
start  of  the  project,  converted  the  existing  learning  objectives 
hierarchies  into  the  form  and  format  specified  by  the  new  DID.  The 
second  process ,  which  occurred  following  lesson  s pec i f  icat ion  deve  1- 
opment,  created  syllabus  oriented  learning  ob jec 1 1 ves  hierarchies  . 

The  data  base  used  for  development  of  the  learning  objective 
hierarchies  was  the  F-14  aircrew  task  Inventory  and  the  set  of  lear¬ 
ning  objectives  which  were  in  existence  at  the  start  of  this  project. 
This  data  base  was  developed  using  an  interactive  review  procedure 
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vith  Navy  F-14  subject  matter  experts.  Individual  and  group  inter¬ 
views  were  conducted  to  control  the  level  of  detail  and  ensure  the 
technical  accuracy  of  the  product.  Sixteen  Instructor  aircrews 
from  VF-124,  VF-1,  VF-2  and  VX-4,  as  well  as  several  VF-124  ISD  per¬ 
sonnel  were  consulted ,  incorporating  an  average  subject  matter  expert 
experience  level  of  10  years  military  service  and  2,000  flight  hours. 

Figure  5  outlines  the  methodology  used  for  revision  of  the 
existing  learning  objective  hierarchy  behaviors,  conditions  and  stan¬ 
dards.  This  procedure  ensured  a  high  degree  of  quality  control  and 
is  outlined  in  the  following  paragraphs.  Special  care  was  exercised 
not  to  omit  any  of  the  data  previously  documented  during  the  revision 
of  the  original  data. 


STEP  1.  Select  a  first  level  objective  from  the  existing  hierarchy. 
This  analysis  began  at  the  top  of  the  hierarchy ,  l.e.,  with  the  ter¬ 
minal  or  upper  level  objectives.  It  was  essential  that  these  objec¬ 
tives  be  the  first  to  be  verified  since  the  remainder  of  the  objec¬ 
tives  were  prerequisite  to  them. 


STEP  2.  Is  there  a  parallel  NFO  objective?  Two  separate  hierarchies 
were  developed,  one  for  the  pilot  and  one  for  the  NFO.  These  hierar¬ 
chies  ,  while  different ,  contained  many  sections  which  were  identical 
for  the  pilot  and  NFO.  By  looking  for  a  parallel  NFO,  or  PILOT, 
objective,  the  work  of  revising  the  objectives  hierarchies  could  be 
substantially  decreased  by  avoiding  duplication. 


STEP  3.  Work  on  both  objectives  together.  If  a  parallel  objective 
existed,  time  was  saved  by  working  on  both  simultaneously. 


STEP  4.  Check  overall  hierarchy  rationale.  This  procedure  involved 
the  analysis  of  a  portion  of  the  existing  pilot  and  corresponding 
NFO  hierarchies  for  validity  of  the  overall  rationale  by  asking  the 
the  following  questions: 

a.  Are  all  necessary  and  sufficient  prerequltles  objectives 
listed? 

b.  Can  any  nlce-to-know  prerequlsities  be  eliminated? 

c.  Are  all  superordinate  and  subordinate ,  dependent  and  inde¬ 
pendent  relationships  correctly  Indicated? 

This  step  ensured  that  all  subtasks,  skills  and  knowledge  re¬ 
quired  to  perform  the  first  order  tasks  were  Included  in  the  hier¬ 
archy.  The  aircrew  task  inventory  was  used  for  the  identification 
of  subordinate  behaviors  by  providing  an  in-depth  job  description 
from  which  these  requisite  tasks ,  skills  and  knowledge  were  derived. 
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Figure  5.  Objectives  Hierarchies  Revision  Process 
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STEP  5.  Is  the  rationale  satisfactory?  In  making  this  decision,  the 
continuously  increasing  knowledge  base  being  gathered  by  the  con¬ 
tractor  was  used.  Each  new  SME  who  worked  with  the  contractor  pro¬ 
vided  valuable  inputs  thus  updating  the  base  of  information. 


STEP  6.  Revise  hierarchy  to  satisfy  rationale.  If  the  decision  in 
Step  5  was  that  the  rationale  was  not  satisfactory,  revisions  were 
made  to  the  hierarchies  and/or  the  objectives  making  up  those  hier¬ 
archies.  In  most  cases  the  hierarchies  proved  to  be  valid,  i.e.,  the 
prerequisite  relationships  were  valid.  The  majority  of  revisions 
were  made  to  the  objectives  themselves.  Once  the  overall  structure 
of  the  hierarchy  was  validated,  attention  was  centered  on  improving 
the  objectives.  They  were  examined  in  terms  of  their  three  parts: 
behavior  statement,  conditions  and  standards. 


The  behavior  statement  is  a  description  of  the  observable  stu¬ 
dent  behavior  which  identifies  that  the  anticipated  learning  has 
occured.  The  behavioral  statements  were  examined  according  to  the 
questions: 
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the  learner  must  react,  the  environment  in  which  the  behavior  should 
occur  and  the  tools  he  may  use. 

The  stimuli  are  the  objects  or  materials  to  which  the  student 
responds.  For  instance,  in  order  to  visually  identify  a  target  the 
student  may  be  given  black  and  white  line  drawings  of  target  aircraft, 
or  color  photographs,  or  even  an  actual  target.  The  specification 
of  stimuli  alone  does  not  unambiguously  determine  conditions.  It 
is  frequently  necessary  to  state,  in  addition  to  the  stimuli,  the 
general  environment  in  which  these  stimuli  are  to  be  presented. 
The  three  primary  types  of  environment  for  the  purposes  of  this  pro¬ 
gram  are  the  classroom,  simulator  and  aircraft.  In  addition  to  the 
specification  of  the  stimuli  and  environment,  the  conditions  should, 
where  necessary,  indicate  what  tools  or  aids  the  learner  may  or 
may  not  employ.  For  example,  the  checklist  for  the  execution  of 
procedures  maybe  allowed  or  disallowed .  In  the  case  of  a  fire  during 
during  engine  start  it  would  be  well  advised  to  disallow  it,  as  the 
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student  should  have  this  committed  to  memory.  In  the  case  of  an  In¬ 
ternal  cockpit  check  It  would  be  advisable  to  allow  it. 

The  quality  of  the  conditions  statement  was  examined  against 
the  following  checklist: 

a.  I 8  the  stimulus  specified? 

b.  Are  the  tools  and  constraints  specified  or  implied? 

c.  Does  the  testing  environment  correspond  to  the  task? 

d.  Has  safety  been  considered? 

e.  Are  security  constraints  considered? 

f.  Has  the  meaning  of  the  behavior  been  kept  consistent  with 
the  conditions? 

The  standards  specify  quality  of  performance  expected  of  the 
student  when  he  has  mastered  the  objective.  Standards  are  defined 
by  factors  affecting  time,  quality,  quantity,  or  a  combination  of 
these.  When  establishing  standards  for  an  objective  the  following 
checklist  was  used: 

a.  Is  the  standard  realistic  and  attainable? 

b.  I s  speed  and/or  accuracy  a  factor? 

c.  Are  there  regulations  which  call  for  specific  standards  of 
performance? 

d.  Is  sequence  important? 

e.  Is  safety  important? 

The  above  process  resulted  In  a  technically  accurate  hierarchy 
of  learning  objectives  for  the  aircrew  members.  However,  it  could 
not  be  assumed  that  these  objectives  were  totally  complete.  As  lesson 
content  was  formulat ed  and  lesson  tryouts  commenced,  new  information 
influenced  the  past  work.  Standards  particularly  were  influenced, 
because  without  tested  instructional  materials  it  was  impossible 
to  pre-determine  them  within  given  constraints.  The  necessity  for 
updates  during  later  stages  of  the  design  process  was  anticipated 
by  the  Data  Item  Description  UDI-H-25717  ,  which  required  update  re¬ 
ports  on  objectives  and  hierarchies. 


STEP  7.  Inspect  next  level  of  objectives.  This  is  the  process  re¬ 
sulting  from  a  YES  answer  to  the  previous  decision  (Step  5).  If 
the  rationale  was  found  to  be  satisfactory  on  the  level  examined, 
then  the  next  level  down  on  the  hierarchy  was  examined. 
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STEP  8.  Are  there  additional  levels?  This  decision  block  slnply  en¬ 
sured  that  the  process  of  determining  the  satisfactory  versus  un¬ 
satisfactory  rationale  of  the  hierarchies  would  continue  until  the 
whole  hierarchy  was  validated  or  revised. 

STEP  9.  Renumber  Hierarchies .  The  final  process  to  be  performed  was 
was  to  verify  the  continuity  of  the  objective  numbering  system.  The 
revised  hierarachies  were  examined  to  ensure  that  additions,  dele¬ 
tions  or  revisions  had  been  properly  accounted  for,  thus  ensuring 
that  the  coding  scheme  used  in  identifying  each  objective  would  con¬ 
tinue  to  be  useful. 

The  process  presented  above  was  used  continuously  during  the 
development  of  the  lesson  specifications.  As  each  of  the  lesson  spe¬ 
cifications  was  developed,  the  level  of  content  became  greater.  This 
Increase  frequently  resulted  in  the  need  for  revisions  to  the  objec¬ 
tives  contained  in  that  lesson  specification.  The  quality  and  stan¬ 
dardization  of  the  objectives  hierarchies  was  maintained  by  the  use 
use  of  this  process. 

The  second  major  process  involved  the  development  of  a  sylla¬ 
bus  oriented  learning  objectives  hierarchy .  This  type  of  hierarchy 
resulted  from  the  processes  of  sorting  and  grouping  the  original 
objectives  into  the  events  of  the  syllabus.  The  mission  phase  ap¬ 
proach  to  the  objectives  hierarchy  was  the  foundation  for  the  struc¬ 
ture  of  the  aircrew  task  listing  and  original  learning  objectives 
hierarchy.  There  were  ten  mission  phases,  consisting  of  1)  mission 
planning,  2)  pre-launch,  3)  takeoff,  4)  navigation,  5)  reconnaissance, 
6)  surveillance,  7)  combat,  8)  in-flight  fundamentals,  9)  approach/ 
landing  and  10)  post  mission  phase.  These  ten  phases  allowed  tracking 
an  aircrew  task  (eventually  a  learning  objective)  throughout  all 
possible  tactical  situations. 

The  first  step  which  occurred  in  syllabus  development  was  the 
sorting  of  the  learning  objectives  according  to  an  aircraft  system 
or  aircrew  function  within  an  area  of  training  in  the  syllabus  .  These 
sorted  objectives  were  then  grouped  into  a  set  of  training  events 
or  lessons.  These  lessons  were  hierarchically  sequenced  (ordered 
simple  to  complex)  for  training.  Within  each  lesson  existed  a  hier¬ 
archy  of  the  specific  objectives  which  were  sorted  into  that  lesson 
from  the  original  mission  phase  objective  hierarchy. 

Lesson  specif lcatlon  development  for  every  syllabus  event,  each 
event  designed  in  the  manner  just  described,  generated  the  requirement 
for  transforming  the  original  hierarchy  of  objectives  to  a  syllabus 
oriented  objective  hierarchy .  Within  each  syllabus  event  was  a  mini- 
hierarchy  of  the  specific  lesson  objectives.  The  syllabus  events 
were  hierarchlcslly  related  themselves  based  on  student  entry  skill 
level.  The  events  were  ordered  simple  to  complex  to  accomplish  the 
terminal  training  objectives.  The  syllabus  oriented  hierarchies 
represented  the  origlnsl  learning  objective  hierarchies,  only  now 
trsnsformed  into  a  format  for  training. 
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The  new  syllabus  oriented  objective  hierarchy  was  designed  to  be 
used  by  the  training  squadron  personnel  in  locating  a  specific  lear¬ 
ning  objective  if  the  process  of  Internal  quality  control  or  the 
results  of  criterion  tests  generated  the  need  for  modifying  or  adding 
learning  objectives. 

Each  objective  within  a  training  event  was  coded  in  a  manner 
that  reflected  not  only  the  order  of  the  objective  in  the  training 
event,  but  also  was  coded  to  the  individual  training  event  in  the 
syllabus.  For  example,  if  a  criterion  test  showed  a  specific  objec¬ 
tive  consistently  falling  criterion,  the  training  personnel  would 
generate  a  requirement  to  examine  that  specific  objective.  Once  the 
objective  was  located  in  the  syllabus  oriented  learning  objective 
hierarchy,  any  required  modification  could  be  made  by  the  appropriate 
personnel  or  educational  specialist. 


MEDIA  SELECTION. 

Duration  of  Task:  September,  1977  -  April,  1978 
Mandays  Expended: 


Program  Manager  0 
Instructional  Psychologist  I  9 
Instructional  Psychologist  II  11 
Instructional  Technologist  I  0 
Subject  Matter  Expert  13 
Clerical  _8 

TOTAL . 41 


The  media  selection  model  used  during  the  F-14  program  develop- 
ment  consisted  of  the  model  specified  in  MIL-T-29053  with  the  addi¬ 
tion  of  a  new  branch  dealing  with  perceptual  motor  sk il  1  s  .  See  Figure s 
6,  7  &  8. 

Each  objective  was  classified  and  given  one  primary  and  two 
alternate  media  choices.  This  process  was  performed  prior  to  the 
grouping  of  objectives  or  the  development  of  syllabus  events.  However, 
economic  and  other  pragmatic  issues  were  not  considered  by  this  or 
any  of  the  other  media  selection  procedures  available  because  in¬ 
structional  concerns  are  invariant  across  training  settings,  while 
economic  constraints  are  not. 

These  economic  constraints  are  particularly  dominant  in  a  pro¬ 
ject  which  is  concerned  with  revising  an  existing  system  for  which 
considerable  media  resources  already  exist.  These  resources  should 
obviously  be  used  where  possible,  therefore,  the  selection  of  media 
will  be  affected  or  biased  by  the  existing  resources. 

In  the  case  of  the  F-14  program,  the  academic  facilities  and 
equipment  were  designed  to  accommodate  lectures  with  visual  support. 
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Individualized  audio-visual  programs  and  Individualized  written  ma- 
terlala.  Regardless  of  the  fact  that  CAI  was  the  logical  media  selec¬ 
tion  for  many  of  the  objectives,  CAI  was  not  an  economically  valid 
choice.  In  such  cases,  the  second  or  third  media  choice  was  used. 

The  existence  of  the  major  training  devices  also  affected  the 
-selection  of  media.  Again,  the  economics  of  adjusting  the  objectives 
to  fit  the  existing  resources  far  outweighed  the  economics  of  modi¬ 
fying  multi-million  dollar  devices. 

The  actual  process  of  media  selection  using  a  given  procedure 
or  "algorithm”  is  tedious  at  best.  Each  individual  objective  is 
treated  separately  and  mentally  processed  through  an  algorithm  which 
determines  a  primary  and  secondary  media  selection. 

There  are  several  reasons  to  doubt  the  efficiency  of  this  entire 
step,  because  of  what  happens  downstream  with  the  media  selected 
for  any  given  objective.  First,  following  the  assignment  of  media 
types,  the  objectives  must  be  grouped  into  lessons.  This  process 
often  results  in  lessons  which  contain  a  variety  of  different  media 
choices.  A  second  media  selec tion ,  or  reorganization,  is  then  neces¬ 
sary  to  reduce  the  number  of  different  media  within  a  lesson  to  a 
number  which  is  ins  true tionally  sound. 

Secondly,  as  the  syllabus  develops,  more  and  more  changes  to 
the  basic  objectives  and  to  the  objective  groupings  take  place. 
Because  of  the  increased  knowledge  base  gained  by  interaction  with 
SME '  8 ,  changes  in  the  tactical  employment  of  the  aircraft  from  the 
time  of  the  original  task  analysis,  etc.  This  In  turn  requires  a 
continuous  reassessment  of  objectives  and  media  selections.  This 
suggests  that  media  selection  should  occur  in  conjunction  with  the 
definition  of  resources  and  training  support  requirements,  rather 
than  earlier  in  the  ISD  process.  This  was  done,  although  neither 
the  MIL  SPEC  or  the  DID  required  that  this  relationship  be  considered. 


TRAINER  MODIFICATION  AND  UTILIZATION  REPORT 

Duration  of  Task:  February,  1978  -  May,  1978 
Mandays  Expended 


Program  Manager  7 
Instructional  Psychologist  I  11 
Instructional  Psychologist  II  0 
Instructional  Technologist  I  5 
Subject  Matter  Expert  36 
Clerical  _5 

TOTAL . 64 


The  trainer  modification  and  u ti 1 i za tion  task  cons  Is  ted  of  five 
steps  aimed  at  defining  needs  for  modifications  to  the  trainers  based 
upon  changes  to  the  training  syllabus.  This  determination  of  the 
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need  for  modifications  was  limited  in  one  respect.  Major  modifica¬ 
tions  to  the  devices  were  not  to  be  recommended.  If  the  trainers 
were  to  be  modified  to  meet  the  requirements  of  the  syllabus,  the 
modifications  would  have  to  be  minor  in  nature. 

The  process  of  determining  the  need  for  trainer  modifications 
.consisted  of  five  steps  as  listed  below  and  shown  in  Figure  9: 


STEP  1.  Compare  syllabus  needs  with  trainer  capabilities.  This  pro¬ 
cess  of  comparison  was  not  performed  exclusively  as  a  part  of  this 
task.  During  the  process  of  media  selection,  the  capabilities  and 
limitations  of  the  training  devices  were  allowed  to  bias  the  selec¬ 
tion  of  the  media  and  as  stated  previously,  major  modifications  to 
the  training  devices  were  not  to  be  recommended  as  a  part  of  this 
effort.  Therefore,  in  many  cases  where  trainer  capabilities  did 
not  fit  requirements  for  training  a  given  objective,  modifications 
were  made  to  the  terminal  objective  or  its  enabling  objectives. 


STEP  2.  Do  the  trainers  meet  the  needs  of  the  syllabus?  A  determin¬ 
ation  was  made  of  the  adequacy  of  the  trainers  to  support  the  syllabus 
being  designed.  As  is  shown,  a  positive  determination  caused  an  end 
to  this  process,  a  negative  determination  led  to  Step  3. 


STEP  3.  Determine  modif ications  needed  to  satisfy  syllabuc  require¬ 
ments  .  In  performing  this  process,  the  objectives  or  event  require¬ 
ments  were  used  in  determing  how  the  training  device  would  need 
to  be  modified.  The  stimulus,  display  requirements,  response  re¬ 
quirements,  performance  monitoring  and  measurement  capabilities  as 
well  as  economic  considerations  were  the  major  determining  factors. 


STEP  4.  Can  the  needs  be  satisfied  with  minor  modifications?  The 
outcome  of  Step  3  was  a  list  of  needed  modifications.  This  list  was 
evaluated  to  determine  the  specific  trainer  changes  desired  and 
the  possibility  of  combining  these  changes  into  a  single  modification. 
Based  upon  the  constraint  that  no  major  modifications  would  be  made, 
a  NO  answer  caused  an  exit  from  this  process,  a  YES  answer  led  to 
Step  5. 


STEP  5.  Submit  Trainer  Equlment  Change  Request  ( TECR )  .  If  the  deter¬ 
mination  was  made  that  the  syllabus  requirements  could  be  satis¬ 
fied  with  minor  modifications,  a  request  (the  TECR)  was  submitted 
to  NAVTRAEQUIPCEN  to  accomplish  that  modification. 
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STUDENT  SYLLABI  (MAS/TARPS). 

Duration  of  Task:  October,  1977  -  May,  1980 
Mondays  Expended: 


Program  Manager 

31 

Instructional  Psychologist 

I 

95 

Instructional  Psychologist 

II 

47 

Instructional  Technologist 

I 

68 

Subject  Matter  Expert 

116 

Clerical 

79 

TOTAL . 436 


The  development  of  the  student  syllabus  proved  to  be  one  of  the 
moat  demanding  and  complex  tasks  performed  during  the  project.  It 
involved  assembling  the  more  than  3500  learning  objectives  which 
had  been  developed  from  the  previously  performed  task  analysis  and 
then,  through  a  series  of  steps,  converting  these  learning  objectives 
into  a  hierarchy  of  lessons.  This  hierarchy  of  lessons  comprised 
the  student  syllabus. 

When  the  task  analysis  was  performed,  a  method  was  devised  to 
ensure  that  each  operator  task  would  be  accounted  for.  This  method 
Involved  the  analysis  of  eight  separate  mission  phases  from  Pre-Flight 
to  Post-Mission  to  determine  what  tasks  were  required  in  each  one 
of  those  eight  mission  phases.  This  method  proved  effective  for 
the  task  analysis  and  the  generation  of  the  objectives  hierarchy 
but  did  not  prove  applicable  to  the  development  of  the  student  syl¬ 
labus.  An  example  of  this  lack  of  applicability  can  be  seen  if  one 
looks  at  the  seventh  mission  phase,  which  is  Approach  and  Landing. 
Landing  the  airplane  is  one  of  the  requirements  the  pilot  must  learn 
prior  to  his  very  first  flight.  If  the  syllabus  were  organized 
along  eight  mission  phases,  landing  would  be  the  next  to  the  last 
thing  that  he  would  learn.  For  this  reason,  the  mission  phase  oriented 
objectives  hierarchy  had  to  be  restructured  into  straining  oriented 
syllabus . 

It  should  be  pointed  out  that  the  retention  of  the  mission  phase 
structure  in, the  development  of  the  objective  hierarchy  leads  to  a 
high  degree  of  redundancy.  As  the  tasks  for  each  phase  in  the  task 
hierarchy  are  transformed  into  behavioral  objectives  and  as  the 
prerequisl tea  for  these  objectives  are  defined  down  to  student  entry 
level,  one  encounters  more  and  more  prerequisites  that  are  common 
across  phases  the  lower  one  descends  on  the  hierarchical  tree.  A  good 
example  la  the  objective*  associated  with  aircraft  systems.  As  one 
might  logically  expec t ,  these  objectives  turn  up  as  prerequisites  to 
any  mission  phase. 

In  developing  the  syllabus  from  the  objective  hierarchies,  the 
first  step  was  a  process  of  sorting  and  grouping  the  learning  objective 
into  the  rough  categories  of  Systems  Operation,  Aircrew  Procedures, 
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Weapons  System  Employment  and  Mission  Support.  Based  on  these  four 
topic  areas,  a  matrix  was  developed  which  contained  subjects  asso¬ 
ciated  with  each  of  the  respective  areas  (See  Figure  10).  Each  topic 
area  was  separated  into  3  phases  representing  increasingly  more 
difficult  subject  areas.  The  learning  objectives  were  then  syste¬ 
matically  sorted  into  these  subject  areas  rather  than  into  the 
mission  phase.  The  simple  to  complex  relationship  was  maintained 
during  this  sorting  process. 

The  next  step  performed  was  the  process  of  les son  development . 
The  objectives  which  had  been  grouped  into  the  sub ject/topic  area 
matrix  were  examined,  one  category  at  a  time.  Each  objective  was 
studied  for  subject  matter  content  as  well  as  inter-relationships 
with  other  learning  objectives  in  the  same  area.  The  objectives  were 
roughly  organized  by  subject  matter  and  learning  types  and  ordered 
from  simple  to  complex  behavior.  Lesson  size  was  determined  by  or¬ 
ganizing  similar  groups  of  objectives  into  what  were  estimated  to 
be  manageable  time  frames.  Finally,  the  lessons  were  grouped  into 
either  academic,  trainer  or  flight  events.  This  process  was  repeated 
until  all  of  the  ob Jec  t i ves  within  the  sub  jec t  /  t opic  area  matrix  had 
been  sorted  and  grouped  into  lessons. 

At  this  point  in  the  syllabus  development  process,  it  was  ne¬ 
cessary  to  identify  instructional  areas  which  would  serve  as  the 
basic  elements  for  the  new  curriculum.  Examples  of  these  broad  in¬ 
structional  areas  are:  aircraft  systems,  basic  flight  maneuvers, 
advanced  combat  maneuvering  and  carrier  qualification  as  shown  in 
Figure  11.  Entry  level  and  exit  skills  were  identified  for  each  of 
these  broad  instructional  areas.  Based  upon  these  entry  and  exit 
skills,  the  lessons  within  these  broad  areas  were  then  defined.  These 
included  lessons  for  the  academic,  trainer  and  flight  phases.  When 
the  entry  level  skills  for  each  of  the  phases  were  prepared,  it  was 
evident  that  the  phases  with  the  lowest  entry  skills  would  be  the 
first  instructional  areas  to  be  presented  to  the  student.  This  pro¬ 
vided  for  the  students  entry  into  the  syllabus  hierarchy.  In  perfor¬ 
ming  this  function,  it  was  determined  that  there  were  multiple  entry 
points  into  the  syllabus  and  multiple  paths  through  this  syllabus 
hierarchy  along  which  the  student  could  progress.  This  hierarchical 
format  allows  for  maximum  ease  in  scheduling  in  that  a  schedules 
officer,  by  looking  at  the  syllabus  hierarchy,  can  determine  alter¬ 
native  events  in  which  the  students  can  validly  participate,  if  the 
resources  for  performing  the  preferred  event  are  not  available. 

A  separate  syllabus  was  developed  for  both  the  pilot  and  the 
NFO  since  it  was  determined  that  the  skills  needed  for  the  Pilot's 
first  flight,  and  for  some  of  his  subsequent  flights,  differed  from 
the  skills  required  for  the  NFO's  first  flight  and  subsequent  flights 
(See  Figure  12).  This  proved  to  be  an  area  of  concern  to  the  Navy 
in  that  it  deviated  from  the  traditional  method  of  parallel  training 
of  pilot  and  NFO;  however,  this  dual  track  type  syllabus  has  resulted 
in  the  elimination  of  high  front  end  loading  and  a  subsequent  higher 
level  of  preparedness  for  the  first  flight  for  both  the  pilot  and 
the  NFO. 
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Figure  10.  Sub jec t/Topic  Area  Matrix 
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TRAINING  SUPPORT  REQUIREMENTS  ANALYSIS  (MAS/TARPS). 

Duration  of  Task:  September,  1977  -  October,  1979 
Mondays  Expended: 


Program  Manager  32 
Instructional  Psychologist  I  86 
Instructional  Psychologist  II  14 
Instructional  Technologist  I  67 
Subject  Matter  Expert  Expert  0 
Clerical  2A 

TOTAL . 252 


The  training  support  requirements  ana  1 y s i s  (TSRA )  was  initiated 
immediately  after  the  generation  of  the  syllabus  and  preceded  the  de¬ 
velopment  of  lesson  formatting  and  segment  specifications.  It  was  pre¬ 
pared  in  accordance  with  the  Military  Specification  and  the  Data 
Item  Description  UDI-H-25722.  These  documents  were  interpreted  in 
the  light  of  the  present  task  which  is  concerned  with  the  redesign 
of  an  existing  training  program. 

An  existing  training  program,  in  contrast  to  a  new  or  emerging 
program  commonly  already  owns  or  has  access  to  a  significant  amount 
of  assets  which,  especially  in  the  case  of  aircrew  training  programs, 
represent  a  considerable  investment.  A  TSRA  for  the  redesign  of  an 
existing  program  is,  therefore,  less  concerned  with  the  generation 
or  procurement  of  new  resources  than  with  the  utilization  and  modi¬ 
fication  of  the  existing  training  assets.  It  is  obvious,  therefore, 
that  a  TSRA  for  the  redesign  of  an  existing  program  must  begin 
with  a  clear  description  of  the  training  task  and  an  inventory  of 
the  existing  training  assets.  The  training  task  is  described  in 
terms  of  the  overall  mission  for  F-14  aircrew  training  and  in  terms 
of  the  general  conditions  under  which  this  training  must  take  place. 
The  two  factors  with  the  greatest  Influence  on  the  type  and  amount 
of  training  assets  required,  the  student  population  and  the  syllabus, 
were  described  and  analyzed  in  detail.  The  assessment  and  inventory 
of  the  current  training  assets  provided  the  baseline  for  the  develop¬ 
ment  of  the  resource  requirements  of  the  proposed  syllabus. 
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The  process  employed  is  depicted  in  Figure  13.  It  consists  es¬ 
sentially  of  an  analysis  phase  and  a  definition  phase.  During  the 
analysis  phase,  three  major  training  system  components  were  assessed: 

a.  the  existing  physical  resources  at  both  squadrons, 

b.  the  personnel,  i.e.  students  and  instructors, 

c.  the  syllabi,  l.e.  the  existing  CNO  syllabi  versus  the 
proposed  syllabi  for  pilots  and  NFO's. 

During  the  definition  phase,  required  resources  were  determined 
for  each  syllabus  event  under  two  different  media  mixes.  The  resource 
requirements  were  then  assessed  over  events  and  daily,  monthly  and 
yearly  requirements  of  each  major  resource  were  determined.  In  ad¬ 
dition,  and  in  variance  from  the  DID,  a  plan  for  a  training  management 
organization  was  developed  in  rudimentary  form.  It  was  felt  that 
it  is  vital  to  include  a  simple  and  effective  training  management 
system  in  any  training  development  procurement  and  that  the  military 
specification  and  the  data  item  descriptions  should  be  amended  ac¬ 
cordingly. 

The  analysis  process  was  relatively  simple  and  straightforward, 
albeit  labor  intensive.  The  various  steps  in  the  process  can  be 
traced  in  Figure  13.  Standard  data  collection  methods  using  observa¬ 
tion,  interviews,  questionnaires  and  document  collection  were  used. 

The  definition  process  was  equally  labor  intensive  and  somewhat 
more  complex.  The  military  spec  1 f ica t ion  did  not  contain  specific 
methods  or  procedures  to  accomplish  this  phase.  The  contractor  was 
therefore  required  to  develop  appropriate  forms  and  procedures. 
These  newly  designed  tools  proved  to  be  very  useful  and  effective 
and  it  is  for  this  reason  that  some  of  them  were  incorporated  in 
the  revised  DID's.  Because  the  resource  definition  process  was  break¬ 
ing  new  ground  it  is  described  in  more  detail  below. 


Procedures  For  Resource  Definition.  Resource  definition  was  ini¬ 
tiated  after  the  analysis  phase  and  after  the  proposed  syllabus  hier¬ 
archy  had  been  developed.  At  this  point,  syllabus  events  for  each 
of  the  three  instructional  environments  were  defined.  For  each  event 
there  was  a  set  of  behavioral  objectives  and  a  lesson  cover  sheet 
with  the  lesson  reference  number,  the  lesson  title  and  a  brief  prose 
description  of  the  purpose  of  the  lesson.  The  lesson  reference  num¬ 
ber  clearly  identified  each  event  by  major  syllabus  area,  specified 
whether  it  was  to  be  administered  to  Pilots  or  NFO's  or  both,  and 
stated  whether  it  was  to  take  place  in  an  academic,  trainer,  or 
flight  environment.  The  goal  was  to  determine  for  each  syllabus 
event  the  specific  media  and  all  other  resources  required  to  support 
this  event,  including  personnel  and  facilities.  In  accordance  with 
the  Military  Specif ication  and  the  Data  Item  Description,  an  attempt 
was  made  to  define  wherever  possible  at  least  two  different  resource 
configurations  for  each  event.  The  guidelines  for  the  definition 
of  the  primary  and  secondary  resource  configurations  were  three 
essentially  independent  variables  which  were  given  different  prior¬ 
ities  as  shown  in  Table  2. 
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Table  2:  Guidelines  for  Resource  Definition 


Priority 

Candidate 

Media  Mix 

No.  1 

No.  2 

1 

Instructional 

Effectiveness 

Cost 

2 

Practicality 

Practicality 

3 

Cost 

Instructional 

Efficiency 

These  variables  represent  the  three  major  concerns  which  must 
be  addressed  during  resource  definition.  All  other  considerations 
such  as  editabillty,  required  stimulus  properties,  existing  media, 
user  acceptance  or  preference,  etc.  are  related  but  subordinate  and 
must  be  dealt  with  In  the  context  of  the  sets  of  priorities  above. 

In  order  to  systematize  the  practical  process  of  resource  de¬ 
finition,  a  lesson  media  and  resource  worksheet  and  a  set  of  proce¬ 
dures  for  its  completion,  were  developed.  The  lesson  media  and  re¬ 
source  worksheet ,  hereinafter  referred  to  as  the  ATF  sheet ,  contains 
a  matrix  for  all  possible  resource  combinations  for  academic,  trainer 
and  flight  events  (See  Figure  14).  The  area  for  academic  Instruction 
was  subdivided  into  four  major  classes:  lecture  events,  self-instruc¬ 
tional  events,  briefings,  and  exams.  For  each  of  these  classes, 
personnel,  media,  facilities  and  evaluation  support  can  be  specified. 
Care  was  taken  to  Include  as  media  those  that  could  be  accomodated 
by  the  instructional  hardware  currently  onboard  at  both  squadrons, 
as  well  as  media  that  might  be  feasible  and  desirable  alternatives. 
The  trainer  area  is  subdivided  into  the  different  types  of  trainers 
that  are  currently  installed  at  both  squadrons  and  the  trainers 
that  are  scheduled  to  be  ready  for  training  during  or  after  program 
implementation.  For  each  of  the  trainers  or  simulators  the  required 
personnel  and  media  support  for  the  training  event  can  be  specified. 
For  flight  events,  the  ATF  sheet  permits  a  specification  of  crew  as 
veil,  as  configuration  for  up  to  four  F-14  aircraft.  Additional  space 
is  provided  to  allow  the  specification  of  facilities,  external  sup¬ 
port,  weather  minima  and  student  flight  hour  minima.  Provisions 
were  made  in  each  of  the  three  areas  to  enable  the  inclusion  of 
resources  and/or  media  which  were  not  explicitly  listed  in  the 
matrix.  Those  entries  can  be  made  under  the  columns  labeled  "other” 
and  particulars  are  explained  under  "remarks"  in  the  middle  section 
of  the  ATF  sheet.  The  labelling  of  the  rows  by  letters  and  the 
columns  by  numbers  permits  coding  of  the  required  resource  support 
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configuration  and  lends  itself,  together  with  the  lesson  reference 
number,  very  easily  to  computer  storage  and  manipulation.  A  separate 
ATF  sheet  was  prepared  for  each  segment  of  each  event.  Trainer  and 
flight  events  commonly  contained  three  separate  segments,  consisting 
of  briefing,  hands-on  portion  or  flight  and  debriefing.  Several 
academic  events  were  mediated  by  two  different  types  of  media.  In 
-this  case  a  different  ATF  sheet  was  prepared  for  each  segment. 

A  set  of  procedures  for  the  completion  of  the  ATF  sheets  (see 
Figures  15  through  20)  was  designed  to  ensure  a  systematic  decision 
making  process  by  all  staff  members  involved  in  this  particular 
evolution.  The  point  should  be  made  that  neither  these  procedures, 
nor  any  other  procedures  for  media  selection  that  may  be  found  in 
the  recent  literature  in  this  area,  can  rightfully  be  called  algor¬ 
ithms.  Algorithms  are  strictly  deterministic  procedures  which  always 
lead  to  the  same  result  given  a  certain  input.  Media  selection  pro¬ 
cedures,  even  though  they  are  non-algor ithmic ,  are  nevertheless 
systematic,  since  they  prescribe  the  type  and  order  of  decisions 
to  be  made  for  the  media  selection  process. 
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Computation  of  Resource  Requirements.  The  completed  ATF  sheets  formed 
the  raw  data  for  the  computation  of  the  resource  requirements  •  These 
computations  were  based  on  the  following  three  assumptions: 

a.  Annual  student  throughput  will  remain  at  FY-78  levels  or 
Increase  very  slightly.  In  FY-78  207  student  inputs  were 
scheduled.  For  purposes  of  this  computation,  an  annual 
student  load  of  220  was  assumed,  allowing  for  a  slight 
margin  of  growth. 

b.  The  distribution  of  students  over  types  and  categories 
will  remain  the  same  as  in  FY-78.  The  following  distribu¬ 
tion  was  assumed: 


Category 

I 

-  602 

Category 

II 

-  272 

Equal  parts  of 

Category 

III 

-  82 

Pilots  and  NFO's 

Category 

IV 

-  42  ' 

Category 

V 

-  12 

c.  Based  on  a  consideration  of  the  backgrounds  of  the  var¬ 
ious  student  categories  as  well  as  of  figures  obtained 
from  the  syllabus  summaries  of  the  current  syllabus,  it 
was  assumed  that  each  student  category  would  receive  the 
following  proportions  of  the  full  syllabus: 

Category  I  receives  1002 

Category  II  receives  802 

Category  III  receives  502 

Category  IV  receives  102 

Category  V  receives  302 

Based  on  these  assumptions,  the  annual  requirements  for  any 
given  resource  can  be  computed  as  follows: 

If 

r  »  amount  of  resource  R  required  to  train  one  stu- 
p  dent  pilot  of  category  i 

i 

S  -  number  of  Category  I  student  pilots  to  be  trained 
p  per  year  “  category  1  student  pilot  load 
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R  “  annual  amount  of  resource  R  required  to  train 
p  all  student  pilots  of  category  1 

1 


then,  for  any  given  category  of  student  pilots: 

R  -  r  S 

P  P  P 

1  11 

and  therefore  the  total  annual  requirement  for  resource  R  for  all 
categories  of  students  together 

1-5 


total 


1-1 


[2]  R  -  r  S  +r  S  +r  S  +r  S  +r  S 

P  PPPPPPPPPP 

total  11  22  33  4  4  55 


The  same  equation  can  be  written  for  NFO's  by  substituting  the 
subscript  n  for  the  subscript  p. 

Given  Assumptions  1  and  2  above,  the  student  load  in  each  cate* 
gory  can  be  expressed  as  a  fraction  of  the  total  annual  student  load 
of  pilots  or  NFO's,  S  or  Sn. 


S  -  .60  S 


S  -  .04  S 
P  P 


S  -  .27  S 
P  1 


S  -  .01  S 


-  .08  S 


*  -  J 
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If  S  -  220 

and  S p-  . 5S 

then  S„  -  110 
P 

and  therefore 

S  -  66.0  ;  S  ■  29.7  ;  S  -  8.8  ;  S  -4.4  ;  S  -1.1 

P  P  P  P  P 

1  2  3  4  5 

then  equation  [2]  becomes 

[31  R  -  66. r  +  29. 7. r  +  8.8.r  +  4.4.r  +  1.1. r 

P  P  P  P  P  P 

total  1  2345 

Given  Assumption  3,  the  resource  requirements  for  training  students 
of  categories  2  through  5  can  be  expressed  as  fractions  of  the  re¬ 
source  requirements  for  category  1  students. 

r  -.8r  ;r  -.5r  ;r  -.lr  ;r  -.3r 

P  PP  PP  PP  P 

2  13  14  15  1 

Substituting  these  values  In  equation  [3]  results  in 

[4]  R  -  66r  +29.7  *  ,8r  +  8.8  *  5r  +  4.4  *  .lr  +  1.1 

P  P  P  P  P 

total  1111 

R  -  9  5r 

P  P 

total  1 

and  by  the  same  reasoning 

R  -  9 5r 

n  n 

total  1 

The  annual  requirement  for  resource  R  for  the  total  student  load  of 

pilots  and  NFO's  together  Is  then 

R  -  R  +  R 

P  n 

total  total 


or 


*  3r 
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R  -  95  (r  +  r 

) 

P 

n 

1 

1 

Annual  requirement  for  resource  R 
for  VF— 101  and  VF-124  combined 


The  computation  of  the  average  daily  requirement  Rq  for  reioure  R  it 
based  on  the  average  number  of  working  days  per  year,  w. 

w  -  50  work  weeks  x  5  -  holidays  -  average  stand  down  days 

w  -  50  •  5-9-2 

w  -  239 

therefore 

R„  -  R  -  95  (r  r  ) 

w  239  p  n 

1  1 

or:  RD-.397(r  +  r  ) 

P  « 

1  1 

rounded  up: 


Average  daily  requirement  for  re¬ 
source  R  for  VF-124  and  VF-101 
combined . 


By  the  same  chain  of  reasoning,  formulas  can  be  derived  which 
allow  the  computation  of  monthly  resource  requirements  at  either 
squadron.  These  formulas  are  based  on  the  assumption  that  60Z  of 
the  student  load  S  will  be  carried  by  VF-124  and  40%  by  VF-101. 


RD  -  .40  (r 

+  r  ) 

P 

n 

1 

1 

Monthly  requirement  for  resource  R 
at  VF-124 


Monthly  requirement  for  resource  R 
at  VF-101 
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Results .  The  TSRA  definition  phase  resulted  in  projections  for  out-  , 

year  support  requirements  which  were  based  on  the  proposed  syllabus  ^ 

as  it  was  defined  at  that  point  in  time.  It  should  be  noted  that 
these  these  projections  changed  as  the  syllabus  underwent  further 
refinement  as  a  result  of  the  development  process  and  user  squadron 
inputs.  These  changed  projections  were  documented  in  update  reports 
.and  the  final  TSRA  report.  The  updating  was  accomplished  with  the 
same  methods  that  led  to  the  initial  projections,  i.e.,  by  means 
of  a  fairly  manpower-intensive  manual  process.  This  experience  showed 
clearly  that  slgnlficang  cost  savings  could  be  obtained,  if  the 
TSRA  process  could  be  supported  by  computer  software,  which  would 
enable  the  development  team  to  not  only  maintain  a  resource  data 
base  keyed  to  syllabus  events  but  also  to  investigate  quickly  the 
results  of  any  syllabus  manipulations  or  resource  changes.  A  data 
base  of  this  nature  could  furthermore  be  used  for  managmement  pur¬ 
poses  over  the  life  cycle  of  the  training  program. 

The  TSRA  resulted  in  specific  recommendations  for  changes  in 
the  available  facilities  and  the  on-board  training  equipment  at  both 
squadrons  as  well  as  in  specific  figures  for  dally,  weekly  monthly 
or  yearly  requirements  of  all  major  resources.  These  resource  re¬ 
quirements  are  shown  below  by  squadron  in  Table  3  and  Table  4. 


f 
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Table  3. 

VF-124  Resource  Requireoents  Summary 


A.  mSOHNEL 


Function 

Total/Type 

Required 

Due  In/ 

On  Hand 

Needed 

1.  Manageaant 

ISD  Model  Nauger 

Staffed 

None 

Media  Manager 

Staffed 

None 

2.  Instructor! 

1*_20  Instructor 
Pilots 

22-31  Instructor 

RIO's 

23  Instructor 
Pilots 

30  Instructor 
RIO's 

2  Instructor 
RIO's 
(alnluua) 

3.  Support 

2  Non-rated  Sailors 

Staffed 

None 

3  Rated  YN  or  DM 
Sailors 

Staffed 

None 

1  Coapug raphlca 
Operator 

Contractor 

Support 

Support  past 
PY81 

1  Educational 
Specialist 

Position 

Vacant 

1  Ed.  Spec. 

B.  I1IUUL/  SUVICES 

Category 

Total/Type 

Required 

Due  In/ 

On  Hand 

Needed 

1.  Student 

Material 

Beproductlon 

2300  pages /student 
*40,000  pgs/yr 

*  notebook  binders/ 
student 

600  blnder/yr 

Hone 

None 

1 

*40,000  pgs / 
yr  1 

600  Notebook 
bl ndera/yr 

S  coab  binders/ 
student 

230  blnder/yr 

None 

730  coab 
blnders/yr 

2.  Student 

Material 

Revision 

ISO  printed  pgs/yr 

1.2  rolls  of 
coupugraphlc  file 
per  year 

Provided  by 
PASOTRACRUPAC 

1.2  rolls  of 
flla/yr 

3.  Media 

Revision 

Services 

up  to  *30  slide 
changes/yr. 

up  to  5  sound/ 
slide  acrlpt  changes/ 

r*’ 

Contractor  support 
through  PY81 

Support  past 
PY81 
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C.  BQUIPMENT 


■•source 


Total/Type  Due  In/ 

Required  On  Hand  Needed 


1. 

Aircraft 

39  sortlca/day 

1050  fit.  hrs/ao . 

25  sorties/ 
day 

775  fit.  hrs/ 
ao. 

14  aortles/day 

275  fit  hrs./ 

BO. 

2. 

Trainers 

NT 

11.3  hra/day 

15.2  hrs/dsy 

None 

3. 

Study 

Carrels 

24  carrels 
(shared  v/VAV  110) 

24  carrels 
equipped 

None 

2.62  hra/day/ 
carrel 

8  hrs/dsy/ 
csrrel 

None 

4. 

16b  Cun- 
sight  Caaera 

8  KS26B 

2 

6 

5. 

Gunsight 

Caaera  FI la 
Projector 

1  Photo  Optical 

Data  Analyser 
(PODA) 

1 

0 

6. 

HUD  Caaera 

15  CTVS  Fairchild 

caaera 

1 

14 

7. 

Video  Recorder 
for  HUD  Caaera 

IS  TRAC  V-1000 

2 

13 

8. 

Video  Playback 
for  above  flla 

1  TRAC  V-4200 

0 

1 

9. 

Mission  Cas¬ 
sette  Recorder 

20  Sony  M-201 

20 

0 

10. 

Overhead 

Projector 

3 

0 

0 

11. 

Videotape 
Caaera  and 
Recorder 

1  AVC-3400 

1 

0 

12. 

Caraaate 

6  Kodak 

2 

4 

13. 

Random  Access 
Slide 

Projector 

3  Kodak  RA-960 

3 

0 

14. 

16aa  Movie 
Projector 

3  Kodak  AV12E6 

3 

0 

IS. 

Study  Carrel 

25 

25 

0 

16. 

3Saa  Slide 
Projector 

25  Kodak  AF-2 

25 

0 

17. 

Audio  Cassette 
Pleyer 

25  VM-752 

25 

0 

18. 

Video  Ceeeettc 
Pleyer 

20  Sony  VP-1000 

20 

0 

1*. 

Color  Monitor 

20  Sony  CVM  1200UA 

20 

0 

2/>. 

Word  Processor 

1  Coapugraphlc 

1 

0 

21. 

Copier 

1  Xerox  9400 

1 

0 

22. 

ATSS  Terminals 

3 

3 

0 
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D.  FACILITIES 


Resource 

Total/Type 

Required 

Due  In/ 

On  Hand 

Needed 

1.  Classrooms 

5  Mediated  Classrooms 

5 

0 

2.  Briefing  rooms 

6 

6 

0 

3.  Study  areas 

400  sq .  f  t . 

225  sq .  ft. 

175  sq.  ft. 

4 .  Ready  room 

1400  sq.  ft. 

1400  sq.  ft. 

0 
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Table  4. 

VF-101  Resource  Requirements  Summary 


Function 

Total/Type 

Required 

Due  In/ 

On  Band 

Needed 

i. 

Manageaent 

Media  Manager 

Staffed 

Hone 

2. 

I oat rue  tort 

13-19  Xnatructor 
Pilots 

21-29  Instructor 

RIO's 

19  Instructor 
Pilots 

17  Instructor 
RIO's 

3  Instructor 
Pilots 
(alninua) 

12  Instructor 
RIO's 
(alninua) 

3. 

Support 

2  Non-rated  Sailors 

Staffed 

None 

B.  MATERIAL/ SERVICES 


Category 

Total/Type 

Required 

Due  In/ 

On  Hand 

Needed 

1 .  Student 
Material 
Reproduction 

2500  p*ge»/itudent 
385,000  pge/yr 

None 

383,000  pgs/ 

yr 

A  notebook  binders/ 
student 

600  blnder/yr 

None 

600  Notebook 
blnders/yr 

3  coab  binders/ 
student 

750  blnder/yr 

None 

750  conb 
blnders/yr 
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C.  BQDIMENT 


Raaource 

Total/Type 

Required 

Due  In/ 

On  Hand 

Heeded 

1. 

Aircraft 

39  aortlea/day 

1030  fit.  hra/mo. 

25  aortic*/ 
day 

775  fit.  hra/ 
mo. 

14  aortle*/d*y 

275  fit  hra./ 
no. 

2. 

Trainer* 

KT 

11.3  hra/day 

15.2  hri/day 

None 

3. 

Study 

Carrel* 

24  carrel* 

(ahared  w/VAW  110) 

24  carrel* 
equipped 

None 

2.62  hra/day/ 
carral 

8  hra/day/ 
carral 

None 

4. 

16an  Cu  fr¬ 
eight  Camera 

8  KB26B 

2 

6 

5. 

Gundght 

Camera  Film 
Projector 

1  Photo  Optical 

Data  Analyzer 
(P0DA) 

1 

0 

«. 

HUD  Camera 

15  CTVS  Fairchild 

camera 

1 

14 

7. 

Video  Recorder 
for  HUD  Camera 

IS  TRAC  V-1000 

2 

13 

8. 

Video  Playback 
for  above  film 

1  TRAC  V-4200 

0 

1 

9. 

Mlaalon  Ca*- 
aette  Recorder 

20  Sony  M-201 

20 

0 

10. 

Overhead 

Projector 

3 

0 

0 

11. 

Videotape 
Camera  and 
Recorder 

1  AVC-3400 

1 

0 

12. 

Caramatc 

6  Kodak 

2 

* 

13. 

Random  Accea* 
Slide 

Projector 

3  Kodak  RA-960 

3 

0 

14. 

16am  Movie 
Projector 

3  Kodak  AV12E6 

3 

0 

IS. 

Study  Carrel 

25 

25 

0 

16. 

35mm  Slide 
Projector 

2S  Kodak  AF-2 

25 

0 

17. 

Audio  Caaaette 
Player 

2S  VM-752 

25 

0 

18. 

Video  Caaaette 
Player 

20  Sony  VP-1000 

20 

0 

It. 

Color  Monitor 

20  Sony  CVM  1200UA 

20 

0 

20. 

Word  Proeeaaor 

1  Conpugraphlc 

1 

0 

21. 

Copier 

1  Xerox  9400 

1 

0 

22. 

ATSS  Terminal* 

3 

3 

0 
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D*  facilities 


Total/Type  Due  In/ 

Resource  Required  On  Hand 


1. 

Classrooms 

3  Mediated  Classrooms 

3 

2. 

Briefing  rooms 

4 

4 

3. 

Study  areas 

300  sq.  ft. 

300  sq .  ft. 

4. 

Ready  room 

1200  sq.  ft. 

1200  sq.  ft 

Heeded 

0 

0 

0 

0 
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STUDENT  TRAINING  COURSES:  LESSON  FORMATTING  AND  SEGMENT  SPECIFICATION. 

Duration  of  Task:  November,  1977  -  December,  1979 
Mandaya  Expended: 


Program  Manager 

99 

Instructional  Psychologist 

I 

414 

Instructional  Psychologist 

II 

205 

Instructional  Technologist 

I 

297 

Subject  Matter  Expert 

809 

Clerical 

576 

TOTAL.  .  .  . 

•  •  • 

.  2500 

The  original  procedure  followed  In  the  development  of  lesson 
specifications  (see  Figure  21)  included  fifteen  steps  and  had  very 
heavy  emphasis  on  early  SME  input  to  ensure  technical  accuracy  and 
completeness  from  the  start.  This  procedure  was  later  revised  when 
it  became  apparent  that  the  necessary  SME  manhours  were  not  available. 
The  revised  procedure  required  less  SME  time  and  more  early  design 
review,  and  resulted  in  equal  or  higher  output  quality  than  the 
original  procedure. 

The  original  lesson  specif icatic n  development  procedure  began 
with  the  contractor  instructional  designer  researching  the  subject 
matter  area.  Following  this,  a  Navy  SME  was  scheduled  to  review  the 
planned  objectives  for  the  lesson  and  to  identify  critical  lesson 
content.  After  meeting  with  the  SME,  the  lesson  content  was  organized 
by  the  instructional  designer  and  reviewed  with  a  contractor  instruc¬ 
tional  psychologist  to  establish  a  lesson  plan.  At  the  completion 
of  this  review,  the  instructional  designer  wrote  generalities  and 
generality  support .  After  the  generalities  and  support  were  written, 
the  lesson  specification  was  again  delivered  to  the  SME  for  review 
and  discussion,  after  which  the  first  draft  was  assembled.  Instruc¬ 
tional  psychologists  reviewed  the  draft  lesson  specification  for 
content  and  instructional  effectiveness.  The  draft  was  revised  and 
typed,  and  again  submitted  to  the  Navy  for  SME  review.  Based  on  the 
Navy's  comments,  the  draft  was  again  revised  and  a  final  copy  delivered 
to  the  Navy  for  SME  and  Navy  ISD  approval. 

This  procedure  was  found  to  be  unsatisfactory  from  two  stand¬ 
points.  First,  Navy  SME’s  were  hard  pressed  for  time,  and  could  not 
meet  the  obligation  required  by  this  development  procedure.  Secondly, 
it  was  found  that  the  instructional  design  Inputs  (primarily  the 
organization  of  the  Instruct lonal  material)  occurred  too  late,  which 
frequently  led  to  major  revisions  at  a  point  where  only  minor  revisions 
should  occur.  For  these  reasons,  the  original  development  procedure 
was  revised  as  shown  in  Figure  22. 

The  revised  procedure  allowed  more  appropriate  utilization  of 
existing  staff  talents  throughout  the  specification  production  pro¬ 
cess.  The  major  change  in  the  process  was  the  addition  of  the  develop¬ 
ment  of  an  outline  which  forced  the  lesson  specification  authors 
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Figure  21.  Original  Lesson  Specification  Developaent  Process 
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Figure  22.  Revised  Lesson  Specif lcstlon  Development  Proce 
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to  concentrate  on  the  structure  of  the  lesson  material  and  permited 
precise  and  early  design  decisions  by  the  instructional  psychologists. 
Again,  the  process  started  out  with  subject  matter  research.  The 
Instructional  designer  obtained,  read  and  studied  appropriate  ex¬ 
isting  material  in  the  NATOPS  manuals,  pocket  checklist,  briefing 
guide,  lecture  guides,  etc.  He  also  viewed  existing  slide  tapes  or 
video  tape  lessons.  If  possible,  he  attended  appropriate  lectures 
or  trainer  events  and  consulted  with  the  NATOPS  officer  for  the  latest 
subject  matter  changes.  He  also  consulted,  at  this  time,  with  the 
Navy  SME  concerning  points  of  ambiguity  of  material.  Then,  after 
he  grouped  and  ordered  his  research  material  on  the  instructional 
topic,  the  author  developed  a  detailed  and  well  structured  outline 
of  the  lesson  or  group  of  lessons  assigned  to  him.  Based  on  this 
outline,  a  lesson  plan  was  developed  which  subdivided  the  lesson 
into  segments,  each  addressing  one  objective.  The  lesson  plan  develop¬ 
ment  included  finalizing  the  lesson  objectives,  structuring  of  the 
generalities  and  determining  number  and  form  of  practice  and  testing 
items.  Following  development  of  the  lesson  plan,  generalities  and 
generality  support  were  written  and  then  reviewed  by  Navy  SME's. 
Following  this  review,  a  complete  lesson  specification  was  assembled 
in  draft  form  and  reviewed  by  an  instructional  psychologist.  The 
draft  was  revised,  typed,  reviewed  again  with  Navy  SME's,  revised 
as  necessary,  and  then  finally  delivered  to  the  Navy.  This  procedure 
required  fewer  Navy  SME  manhours,  reduced  contractor  in-house  re¬ 
visions  and  allowed  for  a  more  efficient  utilization  of  in-house 
SME  capabilities. 

With  this  general  process  overview  as  a  background,  the  fol¬ 
io  wi  ng  pa  rag raphs  provide  a  more  detailed  discussion  of  the  specific 
procedures  and  problems  associated  with  exercising  quality  control 
over  technical  content,  instructional  design  and  formal  standard¬ 
ization. 


Ensuring  Technical  Accuracy  and  Completeness.  Ideally,  the  quest 
for  technical  accuracy  should  begin  with  a  review  of  the  lesson 
objectives  between  the  Navy  SME  and  the  lesson  specification  author. 
This  review  should  yield  finalized  terminal  and  enabling  objectives 
for  a  given  lesson.  This  did  not  work  for  two  reasons.  First,  Navy 
SME's  were  in  very  short  supply,  and  an  attempt  to  schedule  them 
separately  for  this  process  was  found  to  be  inefficient.  Second, 
subject  matter  experts  don't  have  the  requisite  Instructional  tech¬ 
nology  training  to  provide  meaningful  assistance  at  this  point, 
therefore,  the  initial  SME  review  of  technical  input  was  moved  to 
a  point  where  the  SME  had  an  opportunity  to  review  not  only  objectives 
but  also  generalities  and  generality  support  in  written  form  at 
the  same  time.  The  SME  then  had  a  second  opportunity  for  input 
in  step  nine,  where  the  entire  leson  specification  was  available. 
Including  practice  and  testing. 

The  SME  reviews  were  accomplished  in  one  of  two  ways.  One  was 
to  sit  down  face-to-face  with  the  SME  and  to  discuss  the  entire  ma¬ 
terial  point  by  point;  the  other  was  to  leave  the  lesson  specification 
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with  the  SME  end  to  obtain  written  comments.  It  was  found  that  the 
best  choice  was  to  hard-schedule  SME's  for  a  face-to-face  lesson 
review.  This  not  only  provided  Immediate  technical  feedback  but 
It  also  eliminated  a  considerable  amount  of  wasted  time  spent  In 
tracking  down  SHE's  who  were  typically  engaged  In  the  more  urgent 
tasks  of  trying  to  accomplish  the  current  syllabus. 

The  teamwork  between  the  Instructional  designer  or  lesson  author 
and  the  SME  was  generally  a  very  effective.  If  not  always  efficient, 
mechanism  for  producing  technically  correct  lesson  material.  In  a 
situation  where  the  subject  matter  existed  only  In  the  heads  of  the 
SME's,  and  where  frequent  changes  of  procedures  and  hardware  occured, 
It  was  the  only  available  method  of  generating  materials  that  were 
technically  acceptable  to  the  user.  The  fact  that  the  Instructional 
designer  and  the  SME  had  slightly  different  orientations  to  the 
problem  of  designing  new  materials  occasionally  created  obstacles 
which  detracted  from  the  efficiency  of  the  process.  Typical  problems 
were : 

a.  Focusing  the  SME's  critique  on  the  technical  information: 
In  most  cases,  the  SME  for  a  given  topic  was  the  officer 
who  had  taught  this  topic  for  some  time.  In  the  course  of 
his  teaching  the  SME  had  usually  developed  an  Instructional 
strategy  which  differed  from  the  strategy  suggested  by 
the  Instructional  designer,  especially  where  the  topic  was 
taught  in  lecture  form  by  the  SME,  and  was  taught  by  a 
slide/tape  presentation  in  the  new  program.  The  Issues 

of  strategy  then  tended  to  become  the  primary  focus  of  the 
discussion  and  quality  control  of  the  technical  material 
could  only  be  assumed  after  a  clarification  of  the  purpose 
of  the  review. 
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or  unnecessary  material  into  a  given  instructional  event. 
It  was  therefore  Important  that  the  Instructional  designer 
elicited  information  from  the  SME  as  to  why  a  given  piece 
of  content  material  was  needed  by  the  student  and  why  it 
was  needed  at  that  stage  of  training,  and  for  that  objective 
or  behavioral  outcome. 


c.  Obtaining  the  most  experienced  SME:  Ideally,  the  SME  for 
any  area  should  have  been  the  officer  who  was  most  exper¬ 
ienced  (In  spite  of  the  drawbacks  this  entails,  as  discussed 
above).  The  operational  experience  of  seasoned  pilots 
and  NFO's  repreaented  one  of  the  most  valuable  sources  of 
information  available  to  this  project  and  it  was  that  ex¬ 
perience  which  needed  to  be  passed  on  to  the  students. 
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Unfortunately,  the  user  squadron  was  continually  plagued 
with  manpower  problems  and  t  h  a  most  experienced  aircrew 
members  were  the  the  ones  which  were  moat  burdened  with 
collateral  duties  and  thus  least  available. 

d.  Switching  SME ' 8 :  Another  consequence  of  the  user's  manpower 
shortage  was  that  in  some  cases  It  became  necessary  for 
one  SME  to  take  over  a  lessor  from  another  SME  in  midstream. 
This  was  valuable  on  the  one  hand  because  it  provided  a 
second  opinion.  On  the  other  hand,  such  switches  led  to 
some  duplication  of  effort,  repeated  revisions  and  thus 
to  decreased  efficiency  of  the  process. 

In  most  projects  of  this  type,  these  problems  are  probably  un¬ 
avoidable  and  are  intrinsic  features  of  the  process.  It  can  be  as¬ 
sumed  that  some  of  them  could  be  at  lease  partially  solved  by  SME 
training.  However,  for  that  solution  to  be  efficient,  the  SME  train¬ 
ing  would  have  to  be  restricted  to  a  cadre  of  experienced  aircrew 
members  who  for  the  duration  of  the  program,  would  have  no  other 
collateral  duties  except  acting  as  technical  advisors  to  the  con¬ 
tractor.  In  reality  however,  it  is  easier  to  train  an  SME  to  speak 
the  language  of  the  instructional  designer  than  to  turn  the  latter 
into  an  SME,  but  this  training  is  still  time  consuming.  Ideally, 
the  contractor  or  instructional  design  agency,  should  have  its  own 
staff  SME ' s .  Such  a  contractor  thus  has  the  benefit  of  a  rather 
smooth  and  efficient  interaction  with  squadron  SME's.  Under  dif¬ 
ferent  circumstances  however,  an  ISD-trained  SME  cadre  would  seem 
to  be  the  only  really  viable  model  for  ensuring  quality  control 
of  the  technical  material. 


Ensuring  Soundness  of  Instructional  Design.  There  are  four  basic 
criteria  for  determining  the  soundness  of  instructional  design: 

a.  The  objectives  must  be  worthwhile  and  technically  correct. 

b.  There  must  be  an  exact  congruency  between  the  objective, 
the  generality  and  support,  and  the  practice  and  testing. 

c.  There  must  be  a  sound  prerequisite  relationship  between 
objectives  within  the  lesson  and  between  lessons  in  the 
curriculum . 

d.  The  components  of  the  lesson  specification,  that  is  the 
generality,  generality  support,  practice  and  testing  and 
instance  specification,  must  contain  all  required  elements 
and  must  be  in  technically  correct  form. 

Extensive  measures  were  taken  to  ensure  that  the  first  criterion, 
adequacy  of  the  objective,  was  met.  The  lesson  specification  author 
researched  his  assigned  subject  matter  area  using  the  original  ob¬ 
jectives  as  guidelines.  Following  this  research  he  produced  a  de¬ 
tailed  content  outline.  At  this  point,  the  author  and  the  instruc¬ 
tional  psychologist  finalized  the  objectives  to  ensure  that  they 
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are  both  technically  correct  ae  well  as  worthwhile  to  the  instruc¬ 
tional  program.  Experience  has  shown  that  the  objectives  created 
on  the  basis  of  the  task  analysis  are  good  preliainary  or  "working** 
objectives  and  that  they  define  the  doisaln  of  behaviors  to  be  ac¬ 
quired  very  well.  However,  only  detailed  subject  aatter  research 
and  a  good  outline  will  peralt  the  developer  to  focus  precisely  on 
.the  desired  behavior,  conditions  and  standards.  During  this  inten¬ 
sive  and  very  detailed  involvement  with  the  subject  aatter,  it  fre¬ 
quently  becomes  apparent  that  objectives  aust  be  changed  in  some 
way,  l.e.,  either  the  action  statement,  the  conditions  or  the  stan¬ 
dard  must  be  reformulated,  enabling  objectives  must  be  added  or 
deleted,  and  compound  objectives  split  up  into  simpler  objectives. 

During  this  time  of  finalizing  the  objectives  with  the  outline 
in  hand,  the  structure  of  the  generalities  was  es tabllshed  and  prac¬ 
tice  and  testing  was  determined  in  terms  of  type,  method  and  number 
of  items.  The  instructional  psychologist  also  ensured  that  there 
was  an  exact  congruency  between  the  objectives,  the  generality  and 
support  and  the  practice  and  testing.  Finally,  lesson  objective 
hierarchies  were  developed  and  hierarchical  relationships  between 
lessons  were  reexamined.  The  definition  of  the  lesson  plan  was  thus 
the  step  where  all  significant  instructional  design  Inputs  were 
made  and  were  an  initial  quality  check  of  the  first  three  criteria 
(a  through  c  above)  took  place. 

Next,  after  Navy  SHE  review  the  lesson  specification  author 
wrote  generalities  and  supports  and  assembled  a  complete  draft. 
This  draft  was  then  reviewed  by  the  instructional  psychologist  to 
provide  a  final  check  on  criteria  a  through  c  and  to  check  criterion 
d.  Quality  control  at  this  point  was  extensive.  The  contractor 
developed  a  detailed  checklist  for  this  step.  This  checklist  utilized 
a  series  of  questions  to  test  each  component  of  any  given  lesson 
specification.  In  order  to  pass,  all  questions  had  to  be  answered 
positively . 

These  lesson  specification  review  procedures  worked  extremely 
well  and  resulted  in  a  high  and  consistent  output  quality.  The  check¬ 
list  was  not  only  useful  to  the  instructional  psychologist  in  the 
quality  control  process,  but  also  to  the  authors  who  wrote  lesson 
specifications.  Furthermore,  positive  comments  were  received  from 
the  media  developer  following  the  introduction  of  this  document  to 
the  development  process.  The  med ia  developers  found  the  specifica¬ 
tions  to  be  more  clear,  concise,  and  easy  to  follow  when  writing 
scripts  and  slide/tape  presentations. 


Ensuring  Formal  Correctness  and  Standardization.  There  were  four  cri¬ 
teria  for  determining  the  formal  correctness  of  the  lesson  specifi¬ 
cations  . 

a.  All  the  components  are  present  and  in  order. 

b.  Each  component  is  correctly  identified. 
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c  .  Standard  terminology  is  used. 

d.  There  are  no  typographical  errors,  spelling  errors,  etc. 

The  method  for  controlling  formal  correctness  and  standardiza¬ 
tion  consisted  of  four  substeps.  First,  the  checklist  discussed 
previously  was  used  to  determine  that  all  the  parts  were  In  order 
and  that  everything  was  correctly  identified.  The  typist,  after  typing 
the  lesson  specification,  proofread  it  for  typographical  and  spelling 
errors.  It  was  then  given  to  the  author  who  proofread  it  again  for 
technical  accuracy  and  stands'^  terminology.  Finally,  the  Instruc¬ 
tional  psychologist  spotchecked  it  and  verified  the  quality  of  the 
final  product. 


Time  Requirements.  The  time  required  to  insure  the  quality  of  the 
lesson  specifications  was  consistently  higher  than  expected  for 
two  reasons:  the  requirement  of  the  instructional  design  process 
and  the  SME  availability.  First,  the  design  model.  Military  Speci¬ 
fication  MIL-T-29053,  which  was  used  in  this  contract,  required 
lesson  specifications  which  represent  essentially  a  new  design  device. 
Therefore,  there  were  very  few  examples  and  even  less  experience 
to  draw  upon.  Nearly  all  contractor  person  . el  working  on  this  project 
had  to  go  through  an  intensive  learning  process  which  included  both 
formal  training  sessions  and  on-the-job  training. 

The  other  problem  was  Navy  SME  availability.  From  the  begin¬ 
ning,  VF-124  was  overtasked  with  the  requirement  for  providing  tech¬ 
nical  inputs  to  the  lesson  specification  process.  The  contractor 
found  over  and  again  that,  in  spite  of  sincere  squadron  efforts, 
the  SME's  were  just  not  available  and  could  not  be  scheduled  for 
the  number  of  manhours  needed.  SME  participation,  therefore,  was 
consistently  below  the  level  requested.  This  delayed  the  work  out¬ 
put,  not  only  by  making  It  more  time  consuming  to  collect  the  neces¬ 
sary  technical  data  but  also  because  each  hour  given  by  an  SME  re¬ 
quired  an  hour  of  contractor  time.  If  the  SME's  time  had  been  dedi¬ 
cated  more  exclusively  to  this  project,  SME's  could  have  worked 
more  indepentent ly  ,  and  thus  increased  the  output  considerably  with¬ 
out  impact  on  contractor  instructional  developer  time. 


Summary .  Overall,  the  procedures  and  methods  for  quality  control  of 
lesson  specification  authoring  resulted  in  high  quality  lesson  speci¬ 
fications.  Initially,  some  refinements  to  the  procedures  were  neces¬ 
sary  and  extensive  in-house  training  had  to  be  developed.  It  was 
found  that  the  procedures  used  were  workable  both  with  in-house 
lesson  specification  authors  and  with  Navy  SME's.  Time  requirements 
for  quality  control  were  initially  excessive,  but  were  reduced  over 
t  lme  • 
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are  both  technically  correct  as  well  as  worthwhile  to  the  Instruc¬ 
tional  program.  Experience  has  shown  that  the  objectives  created 
on  the  basis  of  the  task  analysis  are  good  preliminary  or  "working" 
objectives  and  that  they  define  the  domain  of  behaviors  to  be  ac¬ 
quired  very  well.  However,  only  detailed  subject  matter  research 
and  a  good  outline  will  permit  the  developer  to  focus  precisely  on 
.the  desired  behavior,  conditions  and  standards.  During  this  inten¬ 
sive  and  very  detailed  involvement  with  the  subject  matter,  it  fre¬ 
quently  becomes  apparent  that  objectives  must  be  changed  in  some 
way,  i.e.,  either  the  action  statement,  the  conditions  or  the  stan¬ 
dard  must  be  reformulated,  enabling  objectives  must  be  added  or 
deleted,  and  compound  objectives  split  up  into  simpler  objectives. 

During  this  time  of  finalizing  the  objectives  with  the  outline 
in  hand,  the  structure  of  the  generalities  was  established  and  prac¬ 
tice  and  testing  was  determined  in  te./ms  of  type,  method  and  number 
of  items.  The  instructional  psychologist  also  ensured  that  there 
was  an  exact  congruency  between  the  objectives,  the  generality  and 
support  and  the  practice  and  testing.  Finally,  lesson  objective 
hierarchies  were  developed  and  hierarchical  relationships  between 
lessons  were  reexamined.  The  definition  of  the  lesson  plan  was  thus 
the  step  where  all  significant  instructional  design  inputs  were 
made  and  were  an  initial  quality  check  of  the  first  three  criteria 
(a  through  c  above)  took  place. 

Next,  after  Navy  SME  review  the  lesson  specification  author 
wrote  generalities  and  supports  and  assembled  a  complete  draft. 
This  draft  was  then  reviewed  by  the  instructional  psychologist  to 
provide  a  final  check  on  criteria  a  through  c  and  to  check  criterion 
d.  Quality  control  at  this  point  was  extensive.  The  contractor 
developed  a  detailed  checklist  for  this  step.  This  checklist  utilized 
a  series  of  questions  to  test  each  component  of  any  given  lesson 
specification.  In  order  to  pass,  all  questions  had  to  be  answered 
posl ti vely . 

These  lesson  specification  review  procedures  worked  extremely 
well  and  resulted  in  a  high  and  consistent  output  quality.  The  check¬ 
list  was  not  only  useful  to  the  instructional  psychologist  in  the 
quality  control  process,  but  also  to  the  authors  who  wrote  lesson 
specifications.  Furthermore,  positive  comments  were  received  from 
the  media  developer  following  the  introduction  of  this  document  to 
the  development  process.  The  media  developers  found  the  specifica¬ 
tions  to  be  more  clear,  concise,  and  easy  to  follow  when  writing 
scripts  and  slide/tape  presentations. 


Ensuring  Formal  Correc  tnes »  and  Standardization.  There  were  four  cri¬ 
teria  for  determining  the  formal  correctness  of  the  lesson  specifi¬ 
cations  . 


a.  All  the  components  are  present  and  in  order. 

b.  Each  component  is  correctly  Identified. 
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C  •  Standard  terainclogy  is  ,sed. 

d.  There  are  no  typographical  errors,  spelling  errors,  etc. 

The  method  for  controlling  formal  correctness  and  standardiza¬ 
tion  consisted  of  four  substeps.  First,  the  checklist  discussed 
previously  was  used  to  determine  that  a  1  i  the  parts  were  in  order 
and  that  everything  was  correctly  identified.  The  typist,  after  typing 
the  lesson  specification,  proofread  it  for  typographical  and  spelling 
errors.  It  was  then  given  to  the  author  who  proofread  it  again  for 
technical  accuracy  and  standard  terminology.  Finally,  the  instruc¬ 
tional  psychologist  spotchect'ed  it  and  verified  the  quality  of  the 
final  produc  t  . 

Time  Requirements.  The  time  roc  aired  to  insure  the  quality  of  the 
lesson  specifications  was  consistently  higher  than  expected  for 
two  reasons:  the  requirement  of  the  instructional  design  process 
and  the  SME  availability.  First,  the  design  model,  Military  Speci¬ 
fication  M1L-T-29053,  which  was  used  in  this  contract,  required 
lesson  specifications  which  represent  essentially  a  new  design  device. 
Therefore,  there  were  very  few  examples  and  even  less  experience 
to  draw  upon.  Nearly  all  contractor  personnel  working  on  this  project 
had  to  go  through  an  Intensive  learning  process  which  included  both 
formal  training  sessions  and  on-the-job  training. 

The  other  problem  was  Navy  SME  availability.  From  c  he  begin¬ 
ning,  VF-124  was  overtasked  with  the  requirement  for  providing  tech¬ 
nical  inputs  to  the  lesson  specification  process.  The  contractor 
found  over  and  again  that,  in  spite  of  sincere  squadron  efforts, 
the  SME's  were  just  not  available  and  could  not  be  scheduled  for 
the  number  of  manhours  needed.  SME  participation,  therefore,  was 
consistently  below  the  level  requested.  This  delayed  the  work  out¬ 
put,  not  only  by  making  it  more  time  consuming  to  collect  the  neces¬ 
sary  technical  data  but  also  because  each  hour  given  by  an  SME  re¬ 
quired  an  hour  of  contractor  time-  If  the  SME's  time  had  been  dedi¬ 
cated  more  exclusively  to  this  project,  SME's  could  have  worked 
more  i nd e pe nt e n t ly  ,  and  thus  inn  eased  the  output  considerably  with¬ 
out  Impact  on  contractor  Instructional  developer  time. 


Summary.  Overall,  the  procedures  and  methods  for  quality  control  of 
lesson  specification  authoring  resulted  in  high  quality  lesson  speci¬ 
fications.  Initially,  some  refinements  to  the  procedures  were  neces¬ 
sary  and  extensive  in-house  training  had  to  be  developed.  It  was 
found  that  the  procedures  used  were  workable  both  with  in-house 
lesson  specification  authors  and  with  Navy  SME's.  Time  requirements 
for  quality  control  were  initially  excessive,  but  were  reduced  over 
t  ime  . 
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In  addition,  some  quality  control  input  was  received  from  the 
media  producer  who  used  the  lesson  specification.  In  general,  such 
input  was  desirable,  since  the  lesson  specification  designer  and 
the  media  producer  are  normally  different  people,  with  different 
requirements  for  utilizing  the  lesson  specification.  These  require¬ 
ments  cannot  always  be  perfectly  anticipated  by  the  lesson  specifi¬ 
cation  developer. 


STUDENT  TRAINING  COURSE  MATERIALS. 

Duration  of  TaBk:  May,  1978  -  May,  1980 
Mandays  Expended: 


Program  Manager  35 
Instructional  Psychologist  I  157 
Instructional  Psychologist  II  234 
Instructional  Technologist  I  166 
Subject  Matter  Expert  838 
Clerical  483 

TOTAL . 1913 


Prototype  Production  of  Lesson  Materials  and  Tests.  Prior  to  large- 
scale  development  of  the  student  training  materials,  the  formats 
for  each  type  of  course  material  were  developed  and  finalized.  This 
process  (see  Figure  23),  began  with  the  selection  of  one  prototype 
lesson  specification  for  each  media  type.  While  there  were  no  hard 
and  fast  criteria  for  the  selection  of  the  topic  of  each  of  these 
prototype  lessons,  there  were  some  considerations.  One  consideration 
was  the  location  of  the  lesson  within  the  syllabus,  and  another,  to 
what  degree  this  lesson  was  representative  of  the  lessons  to  be  pro¬ 
duced  in  that  type  of  medium. 

The  second  step  in  the  process  was  to  develop  a  single  format 
based  on  the  type  of  medium  developed.  There  were  several  questions 
that  drove  the  design  of  each  of  the  media  formats.  These  questions 
wer e : 

a.  How  can  the  gener a  1 i t y  be s t  be  highlighted  in  this  type  of 
med i urn  ? 

b.  How  can  practice  and  feedback  be  most  effectively  integra¬ 
ted  through  this  type  of  medium? 

c.  What  is  the  clearest,  most  straightforward  way  to  organize 
material  within  this  medium  for  optimal  Instructional  effec¬ 
tiveness? 

d.  How  can  the  format  lend  itself  to  easy  access  of  information 
on  the  part  of  the  student  at  a  later  date? 
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c.  How  should  the  lesson  parts  be  sequenced  for  optimal  effec¬ 
tiveness  ? 

Once  each  of  the  formats  was  developed,  it  was  presented  to  the 
Navy  for  approval.  The  formats  were  generally  reviewed  by  a  small 
group,  in  a  conference  setting.  The  group  usually  included  four  or 
five  Navy  personnel,  including  the  ISD officers,  SME  '  e  and  consultants 
from  NPRDC.  Once  the  format  was  agreed  upon,  it  was  tried  out  on 
a  small  group  of  students.  In  most  cases,  it  was  difficult  for  the 
student  to  give  much  input  about  the  usefulness  of  the  formats  since 
they  had  no  basis  for  comparison.  Conversely,  the  Instructors  always 
had  input  which,  understandably,  was  usually  concerned  wi  th  the  ease 
of  use  of  the  format.  Usually  any  given  prototype  event  went  through 
several  revisions  before  it  was  considered  ready  for  large  scale 
production.  As  the  flow  chart  shows,  provisions  were  made  to  permit 
format  changes  even  after  the  start  of  large  scale  production. 


Training  Contractor  Authors.  Once  the  formats  were  finalized,  con¬ 
tractor  authors  were  trained  as  to  what  would  be  contained  under 
each  of  the  major  headings  in  each  format.  For  example,  in  the 
case  of  trainer  briefing  guides,  there  were  headings  such  as:  training 
exercises,  training  objectives,  reference  material,  etc.  There  was 
also  a  heading  called  "reference  material"  contained  in  flights. 
All  authors  needed  Job  aids  as  well  as  training  sessions  to  be  able 
able  to  produce  high  quality,  standardized  course  material  components 
under  each  heading.  Each  author  spent  approximately  one  day  reading 
the  reference  materials  and  producing  practice  components.  He  re¬ 
ceived  feedback  from  in-house  experts  and  then  proceeded  with  an 
OJT  phase  over  the  next  two  weeks.  During  the  OJT  phase,  his  output 
was  continuously  checked  and  feedback  was  provided  as  frequently 
a 8  possible.  After  the  OJT  phase,  normal  quality  control  procedures 
were  employed. 


Large  Scale  Production.  One  general  overall  procedure  was  used  for 
the  development  of  all  lesson  materals  (see  Figure  24).  The  first 
step  in  this  procedure  began  with  a  Navy  approved  lesson  specifica¬ 
tion,  that  contained  all  the  component  parts  necessary  for  authoring 
the  lessons,  such  as  objectives,  generalities,  generality  support, 
instances  and  practices,  testing  and  examples. 

Once  the  author  had  the  approved  lesson  s pec i f icat ion ,  he  gath¬ 
ered  all  pertinent  reference  material  that  might  be  helpful  in  auth¬ 
oring  the  lesson  material.  This  step  was  required  primarily  when 
the  lesson  specification  (LSP)  author  was  not  also  the  training 
materials  author.  Even  though  the  LSP  should  have  satisfied  all  in¬ 
formational  needs,  it  did  not  and  could  not  answer  all  the  technical 
questions  that  an  author  might  have  when  translating  the  LSP  infor¬ 
mation  into  smooth  discourse. 
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Next,  the  objectives  were  reviewed  again.  In  some  cases,  changes 
In  action  verbs ,  criteria  and  standards  were  suggested  by  the  subject 
■atter  on  the  one  hand  and  by  practical  considerations  on  the  other 
hand . 


Next,  the  lesson  was  written  out  by  hand.  This  was  the  most 
lengthy  port  ion  of  the  authoring  process,  and  It  frequently  required 
additional  Interface  among  contractor  and  Navy  SME * s . 

Once  the  first  draft  was  written  by  hand,  contractor  ISD  spe¬ 
cialists  reviewed  the  lesson  for  Internal  consistency,  format  and 
Instructional  effectiveness.  At  this  point,  the  lesson  was  typed 
in  the  first  rough  draft. 

It  should  be  mentioned  at  this  point,  that  often,  toward  the 
end  of  the  project,  the  lesson  materials  were  reviewed  by  Navy  SME's 
In  the  handwritten  form.  This  Is  particularly  true  of  lecture  guides. 
This  Intermediate  step  was  taken  to  prevent  major  revisions  of  typed 
materials . 

Once  the  lesson  was  typed,  it  was  sent  to  the  Navy  and  reviewed 
by  both  the  ISD  officer  and  the  Navy  SME  for  that  particular  subject 
matter  area.  If  major  revisions  were  required,  there  was  usually 
a  meeting  between  the  SME  and  the  contractor  author,  during  which 
the  revisions  were  defined.  The  lesson  was  then  rewritten  and  sent 
back  to  the  Navy  for  review. 

At  this  point,  after  the  Navy  approved  the  corrections,  the 
material  would  go  through  a  small  group  tryout  of  a  form  dependant 
upon  the  medium. 

Once  the  event  had  been  tried  out,  Navy  &  contractor  SME ' s  and 
Instructional  designers  made  decisions  about  how  the  lessons  might 
be  altered  to  Improve  them  and  make  them  more  effective.  Depending 
upon  the  kinds  of  revisions  that  were  required,  the  materials  either 
went  through  another  authoring  process,  or  were  revised  on  a  small 
scale,  and  then  put  into  final  production. 

The  final  production  process  was  also  a  fairly  labor-intensive 
procedure.  It  Involved  a  great  deal  of  editing  and  review  of  final 
copy  for  typos,  format,  type  sizes,  etc.  It  required  at  least  three 
people  to  proof  the  final  copy:  an  editor,  an  SME,  and  the  production 
supervisor . 

Once  the  final  copy  was  complete,  it  was  delivered  to  the  Navy 
and  subsequently  approved  by  Navy  ISD  personnel.  This  completed  the 
first  full  cycle  of  production  of  materials,  after  which  time  the 
material  was  Implemented  and  used  by  the  first  class.  Once  the  first 
class  was  through  using  sections  of  the  Instructional  materials, 
the  revision  process,  which  entailed  an  entirely  new  cycle  of  re¬ 
visions  and  corrections,  began. 
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QUALITY  CONTROL  PLAN  &  REPORT. 

Duration  of  Task:  November,  1977  -  October,  1979 
Mandays  Expended: 


Program  Manager  8 
Instructional  Psychologist  I  10 
Instructional  Psychologist  II  36 
Instructional  Technologist  I  30 
Subject  Matter  Expert  3 
Clerical  _8 

TOTAL . 95 


This  section  outlines  the  procedures  used  for  quality  control 
during  the  accomplishment  of  the  project.  Figure  25  depicts  the 
entire  quality  control  process  including  formative  evaluation,  in¬ 
ternal  quality  control,  and  external  quality  control.  Outyear  quality 
control  will  not  be  a  subject  of  this  section. 

The  quality  control  procedures,  while  very  complex,  were  not 
unlike  the  general  procedures  followed  in  most  traditional  ISD  pro¬ 
jects.  For  purposes  of  simplification  and  summarization,  the  process 
has  been  reduced  to  eleven  steps. 


STEP  1:  Input.  In  the  development  of  the  lesson  materials  for  the 
F-14  Aircrew  Training  Program,  the  quality  control  procedure  began 
with  the  following  inputs:  lesson  specif ications ,  draft  of  learning 
materials  and,  blank  data  collection  forms.  During  the  lesson  speci¬ 
fication  stage,  there  were  several  quality  control  reviews  undertaken, 
both  i ns t rue t ional ly  and  technically.  That  is,  instructional  de¬ 
signers  and  subject  matter  experts  reviewed  lesson  specifications 
before  authoring  of  the  materials  for  instructional  soundness  and 
technical  accuracy. 


STEP  2:  Instructional  Review.  Once  the  i n s t r uc t 1 ona 1 ma t er ial s  were 
drafted,  the  first  person  to  review  and  make  suggestions  for  revision 
was  the  senior  instructional  psychologist  who  provided  input  on  the 
first  several  drafts  of  each  individual  media.  Once  formats  had  been 
established,  the  process  was  then  turned  over  to  an  instructional 
technologist  who  ensured  that  lesson  materials  adhered  closely  to 
lesson  specifications,  were  i ns t rue 1 1 onal ly  sound,  and  were  readable, 
clear,  and  concise. 


STEP  3:  Technical  Review.  Once  the  lesson  author  received  sugges¬ 
tions  for  revision  from  the  instructional  psychologist,  he  revised 
materials  and  submitted  them  to  a  Navy  subject  matter  expert.  The 
Navy  SME  in  that  phase  area  reviewed  the  instructional  materials 
draft  prior  to  t ryout .  Revisions  to  the  Instructional  materials  at 
this  point  were  coordinated  through  the  contractor  lesson  author. 
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STEP  1  Input: 
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Figure  25.  Quality  Control  Proceaa 
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STEP  4s  Individual  Tryouts.  When  time  permitted,  individual  tryouts 
were  conducted  with  all  lesson  materials.  More  often  than  not,  how¬ 
ever,  individual  tryouts  were  only  conducted  with  texts  and  tests 
due  to  severe  time  limitations.  The  individual  tryouts  were  conducted 
relatively  informally  by  in-house  contractor  SME ' s  who  assumed  the 
role  of  students. 


STEP  5:  Acceptable?  After  the  individual  tryouts,  the  most  apparent 
or  obvious  problems  were  identified,  and  materials  were  then  revised 
accordingly . 


STEP  6;  Small  Group  Tryout.  Small  group  tryouts  were  conducted  for 
all  replicable  events.  Slide  tapes  were  the  largest  single  medium 
evaluated  by  this  method.  The  groups  varied  in  number  from  four 
to  six  current  FRS  students.  Preand  post  testing  was  also  conducted 
and  student  attitude  questionnaires  were  completed  on  each  lesson 
tried  out . 

The  final  test  scores  of  the  small  group  together  with  the  in¬ 
formation  obtained  from  the  student  attitude  Inventory  resulted  in 
another  revision  to  the  instructional  material.  Once  this  revision 
was  completed,  the  materials  were  then  considered  ready  for  imple¬ 
mentation  . 


STEP  7:  Large  Group  Tryout.  The  large  group  tryouts  occured  during 
Implementation  since  time  and  student  availability  made  these  tryouts 
infeasible  prior  to  implementation.  Again,  test  data  was  analyzed, 
and  student  and  instructor  attitude  questionnaires  were  used  as 
the  basis  for  revisions.  Most  slide  tapes  were  of  pre-production 
quality  during  these  tryouts  and  did  not  go  into  final  production 
until  after  implementation. 


STEP  8;  Production.  During  this  step  of  the  quality  control  proce¬ 
dure,  the  final  materials  were  reviewed  by  an  Instructional  designer 
for  instructional  quality,  a  con t r a c t or  SME  for  technical  accuracy, 
a  technical  editor  for  clarity,  conciseness  and  typing  accuracy, 
and  finally  by  the  Navy. 


STEP  9;  Integration.  Once  materials  were  completed  in  final  form, 
they  were  integrated  into  the  syllabus. 


STEP  10:  Revision.  Materials  were  monitored  very  closely  through 
student  test  results  and  printed  questionnaires.  Both  instructional 
psychologists  and  technologists  and  SME’s  regularly  reviewed  ques¬ 
tionnaires  and  made  updates  and/or  revisions  to  the  instructional 
materials  as  required. 
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STEP  11:  Meet  Fleet  Meeds?  Detailed  plans  have  been  described  In 
earlier  reports  for  collecting  data  from  the  fleet  regarding  graduate 
performance  and  adequacy  of  the  course  In  meeting  fleet  needs.  Most 
of  this  data  will  be  collected  continuously  by  questionnaires  and 
Interviews  after  the  graduates  have  been  In  an  operating  squadron  long 
enough  to  demonstrate  performance. 

To  summarize,  the  quality  control  procedure  in  the  development 
of  lesson  materials  can  best  be  described  as  an  Iterative  process. 
At  several  points  during  the  development  of  the  materials,  they  were 
reviewed  by  SHE ' s ,  instructional  designers,  and  by  typical  students . 
Before  implementation,  the  students  interacted  with  the  materials 
as  though  the  materials  were  part  of  the  current  syllabus.  As  a 
result,  data  on  practice  and  feedback  items  and  test  scores  were 
used  to  revise  the  materials.  This  was  accomplished  so  that  as  the 
material  became  a  permanent  part  of  the  syllabus,  they  would  be  In 
the  best  form  possible.  Plans  have  been  made  for  lesson  materials 
to  be  continuously  subject  to  review  and  reassessment  throughout 
the  life  cycle  of  the  training  program. 


IMPLEMENTATION  PLAN. 


Duration  of  Task:  September,  1977  -  October,  1979 
Mandays  Expended: 

Program  Manager  28 

Instructional  Psychologist  I  39 

Instructional  Psychologist  II  63 

Instructional  Technologist  I  34 

Subject  Matter  Expert  8 

Clerical  12 

TOTAL . 184 


The  F-14  ISD  project  involved  replacing  one  training  system 
with  another  training  system.  Since  this  replacement  had  to  be  accom¬ 
plished  without  stopping  the  constant  flow  of  graduates,  the  imple¬ 
mentation  of  the  revised  training  system  required  very  careful  plan¬ 
ning.  This  section  discusses  the  development  of  the  implementation 
plan.  The  steps  are  represented  in  a  detailed  flow  diagram  In 
Figure  26. 


predominant  instructional  strategies)  of  the  existing  and  new  training 
systems  were  compared  wl th  each  other .  Most  of  the  data  required  for 
this  comparison  was  contained  In  the  Training  Support  Requirements 


STEP  1:  Compare  the  Proposed  and  the  Existing  Training  Systems. 
In  this  step,  the  principle  features  (the  length,  type  of  sequencing, 
degree  of  individualization,  degree  of  self-pacing,  media  mix  and 


Figure 
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STtP  1. 


26.  Implementation  Plan  Developne 
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Analysis  (TSRA) .  The  result  of  this  step  was  a  syllabi  Identifica¬ 
tion  of  the  factors  which  would  affect  Implementation.  These  results 
provided  Inputs  to  the  development  of  the  plan  for  Implementation 
of  the  new  syllabus. 


■STEP  2:  Describe  Ongoing  Training  and  Management  Procedures.  The 
task  of  Implementation  of  the  proposed  F-14  program  was  facilitated 
by  the  existence  of  an  ongoing  training  program.  It  was  determined 
that  the  existing  procedures  for  administration,  student  management , 
and  facilities  usage  were  acceptable  for  implementing  the  proposed 
system.  This  decision  was  reached  through  a  series  of  interviews 
with  Navy  personnel  in  conjunction  with  in-house  expertise.  These 
results  determined  the  areas  of  the  existing  system  needing  detailed 
plans  and  procedures  for  management. 


STEP  3:  Determine  Integration/Implementation  Schedule.  This  step 

established  the  major  events  to  take  place  in  setting  the  stage  for 
Implementation  of  the  proposed  system.  All  the  required  activities 
were  shown  on  detailed  tine  lines  arranged  to  meet  the  lmplementat ion 
schedule.  The  major  milestones  were: 

a.  Course  Material  Development 

b.  Instructor  Training 

c.  Management  Procedures 

d.  Internal/External  Quality  Control 


STEP  4:  Define  Implementation  Constraints  and  Contingencies.  The 
ideal  implementation  of  a  training  program  assumes  that  all  compo¬ 
nents  of  the  system  are  available  for  employment  at  the  time  of  im¬ 
plementation.  Unfortunately,  it  was  determined  in  the  preceding 
step  that  Implementation  would  occur  with  less  than  the  optimum 
conditions  existing.  Many  components  would  not  be  ready  for  imple¬ 
mentation,  and,  therefore,  neither  would  the  final  phase  of  Instructor 
training  which  required  those  components .  As  a  result,  the  contractor 
and  the  Navy  developed  a  contingency  plan  for  implementation  which 
would  reduce  the  ramlf icatlona  of  these  shortcomings. 


STEP  5:  Develop  Management  Plan.  As  a  result  of  step  2,  which  de¬ 
fined  existing  training  and  management  procedures,  the  various  de¬ 
tailed  plans  for  organization  and  management  of  the  components  of 
the  proposed  system  were  developed.  These  plans  were  reviewed  In 
draft  form  by  the  Navy  and  then  used  to  develop  the  final  Implemen¬ 
tation  plan.  These  sub-plans  are  described  In  the  discussion  that 
follows . 
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a.  Equipment  Storage,  Maintenance  and  Facilities  Utilization 

Plan . 

This  plan  detailed  the  procedures,  schedules,  and  personnel 
roles  and  responsibilities  involved  in  equipment  maintenance 
and  storage,  as  well  as  facilities  utilization.  Both  training 
squadrons  were  considered  separately,  and  the  data  was  documented 
in  the  TSRA  and  the  subsequent  update®  to  the  TSRA.  These  pro¬ 
cedures  were  essentially  in  operation  at  each  respective  squa¬ 
dron,  and  no  changes  were  made  to  the  existing  mode  of  operation. 

b.  Student  Management  Plan. 

This  plan  also  used  the  existing  procedures  and  schedules 
in  operation  within  each  training  squadron.  It  was  felt  that 
there  was  no  change  needed  to  these  procedures,  especially 
since  greater  support  from  ATSS  was  becoming  available  to  the 
squadrons.  The  only  unknown  was  the  student  flow  rate  which 
was  not  controllable  by  the  squadrons. 

c.  Training  Materials  Management  Plan. 

The  management  plan  established  the  detailed  procedures 
and  responsibilities  for  storage ,  issue,  reproduction  and  main¬ 
tenance  of  all  training  materials.  Included  here  were  the  pro¬ 
cedures  and  the  schedule  for  the  Navy  and  the  contractor  to 
complete  final  production  of  all  student  materials  four  weeks 
prior  to  implementation.  Also  included  were,  the  procedures 
for  internal  quality  control  of  the  training  materials  by  Navy 
personnel.  Based  on  this  schedule,  the  plans  for  instructor 
training  were  then  developed. 

d.  Instructor  Management  Plan. 

This  plan  established  the  policies  and  procedures  regarding 
the  utilization  of  instructors  within  the  instructional  system. 
The  plan  specified  instructor  roles  and  responsibilities  with 
respect  to  students,  materials,  syllabus  requirements,  testing, 
monitoring,  counseling,  and  instruction.  There  were  two  types 
of  instructor  training  to  consider:  first,  training  the  in¬ 
structors  onboard  prior  to  implementation,  and  second,  training 
newly  ass igned  ins t r uc t or s ,  including  ins t rue t lona 1  management 
personnel,  after  the  program  was  implemented.  With  Navy  agree¬ 
ment  on  these  plans,  a  detailed,  time-lined  schedule  for  in¬ 
structor  training  was  developed  which  required  a  start  date 
four  weeks  before  implementation. 

e.  Syllabus  Scheduling  and  Resource  Management  Plan. 

This  plan  detailed  the  procedures  and  techniques  for  use  in 
scheduling  all  resources  so  that  they  interfaced  properly  with 
the  syllabus  sequence  requirements.  The  contractor  provided 
training  in  the  use  of  manual  scheduling  aids,  and  also  pro¬ 
vided  ADP  compatible  data  for  ATSS  syllabuB  management.  The 
squadrons  were  able  to  adapt  the  requirements  of  the  proposed 
syllabi  scheduling  to  the  existing  procedures  for  daily  resource 
management.  The  contractor  provided  a  master  training  course 
schedule  for  planning  purposes  ,  as  well  as  a  unique  hierarchical 
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syllabus  layout  to  enable  instructlonally  valid  alternatives 
to  a  dally  scheduling.  This  type  of  layout  was  developed  to  allow 
for  contingency  planning  based  on  daily  changes  and  const ralnts 
in  resource  availability.  The  existing  procedures  governing 
identification  and  counseling  of  students  with  problens,  and  re¬ 
cycling  of  students  through  test  points  were  reviewed  and  remain 
unchanged  for  Implementation. 

f.  Student  Grading  and  Evaluation  Plan. 

This  plan  established  the  procedures  for  grading  and  quali¬ 
fication  of  students.  It  specified  the  academic  test  format 
and  trainer/ flight  grade  sheet  format  that  was  to  be  used  to 
determine  the  pass/fail  or  numerical  grade  for  each  tested  event. 
The  instructor  training  materials  detailed  the  procedures  for 
training  the  instructors  in  the  use  of  these  materials.  The 
plan  also  addressed  the  requirement  for  an  Instructor  on  duty 
in  the  learning  center  and  his  defined  roles  and  responsi¬ 
bilities. 


STEP  6:  Develop  the  Management  Organisation  and  Functions.  An  anal¬ 
ysis  of  management  requirements  was  performed  by  the  Navy  with  support 
from  the  contractor.  This  analysis  resulted  in  recommendations  for 
a  change  in  the  organisation  of  the  Operations  department  in  order 
to  Increase  the  emphasis  on  training.  These  organizational  changes 
were  made  prior  to  implementation  so  that  the  new  organizational 
elements  and  the  lines  of  communication  existed  at  the  time  of 
Implementation . 


STEP  7;  Develop  Final  Implementation  Plan.  Based  upon  the  previous 
steps,  the  final  Implementation  plan  was  developed.  It  consisted 
of  schedules  for  the  processes  involved  in  implementation.  These 
processes  Included  the  coordination  of  delivery  and  storage  of  train¬ 
ing  materials,  schedules  for  the  remainder  of  formative  evaluation, 
internal  quality  control,  and  student  scheduling. 
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INSTRUCTOR  TRAINING  COURSE  MATERIALS. 

Duration  of  Task:  April,  1978  -  December,  1979 
Mandays  Expended: 


Program  Manager 

11 

Instructional 

Psychologist  I 

21 

Ins  tructional 

Psychologist  II 

76 

Ins  truct ional 

Technologist  I 

5 

Subject  Matter 

Expert 

58 

Clerical 

9 

TOTAL . 180 

Two  deliverables,  instructor  training  course  materials  and 
training  device  instructor/operator  training  materials,  were  combined 
into  one  in  order  to  streamline  the  documentation  since  the  training 
device  instructors  also  taught  academics  and  flights. 

Jn  both  cases,  these  materials  were  developed  through  the  use  of 
the  same  ISD  methodology  used  in  the  development  of  the  student 
syllabi.  The  tasks  to  be  performed  by  the  instructors  were  analyzed, 
objectives  were  developed  from  those  tasks,  and  a  training  syllabus 
was  developed  from  the  learning  objectives.  The  purpose  of  these 
materials  was  to  inform  the  squadron  instructors  of  the  ISD  method¬ 
ology  that  was  used  in  developing  their  syllabus  and  to  train  them 
in  the  use  of  the  student  training  materials  that  were  produced 
as  a  result  of  this  project. 

The  materials  were  developed  primarily  by  the  contractor,  how¬ 
ever,  there  was  significant  interaction  with  senior  SME's  at  VF-124. 
In  particular,  the  officer  in  charge  of  Instructors  Under  Training 
(IUT)  and  standardization  was  used  as  the  primary  source  of  infor¬ 
mation  and  as  the  primary  reviewer  of  the  materials  produced. 

There  were  significant  interrelationships  between  this  task  and 
others  performed  during  this  project.  The  instructor  training  course 
materials  described  the  revised  syllabus,  the  use  of  the  new  training 
materials,  the  revised  management  organization  and  the  role  of  the 
instructor  in  the  process  of  internal  quality  control.  Any  changes 
which  occurred  in  these  four  tasks  would  have  had  an  effect  on  the 
instructor  training  materials.  For  this  reason,  the  final  production 
of  these  materials  occurred  late  in  the  project. 

The  training  device  instructor/operator  training  materials  were 
not  subject  to  the  effects  mentioned  in  the  previous  paragraphs. 
These^ materials  were  designed  to  teach  a  new  instructor  to  operate 
the  training  devices  without  the  need  for  a  tutor.  In  the  past,  in¬ 
structors  had  been  trained  in  a  one  on  one  environment  with  a  quali¬ 
fied  Instructor,  however,  this  was  determined  to  be  an  area  where 
self  instruction  could  achieve  a  reduction  in  the  instructor  workload 
with  no  reduction  in  training  effectiveness.  The  materials  produced 
were  designed  to  be  partially  pre-read  after  which,  hands  on  exercises 
at  the  device  console  were  to  be  performed. 
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TRAINING  PROGRAM  WORK  REPORT. 

Duration  of  Task:  February,  1980  -  June,  1980 
Mandays  Expended: 


Program  Manager  5 
Instructional  Psychologist  I  11 
Instructional  Psychologist  II  0 
Instructional  Technologist  I  2 
Subject  Matter  Expert  8 
Clerical  _4 

TOTAL . 30 


This  task  was  concerned  solely  with  the  writing  of  the  Training 
Program  Work  Report. 


TASK  LISTINGS  (TARPS). 

Duration  of  Task:  August,  1979  -  June,  1979 
Mandays  Expended: 


Program  Manager  4 
Instructional  Psychologist  I  27 
Instructional  Psychologist  II  41 
Instructional  Technologist  I  24 
Subject  Matter  Expert  2 
Clerical  5_5 

TOTAL . 15$ 


While  performing  the  task  listings  for  the  TARPS  syllabus,  a  pro¬ 
blem  common  to  all  emerging  systems  was  encountered.  The  problem 
was  that  no  Subject  Matter  Experts  in  the  use  of  the  RF-14  TARPS 
Reconnaissance  system  existed.  The  TARPS  pod  was  undergoing  develop¬ 
ment  at  the  time  the  task  listings  were  required,  therefore,  the 
method  which  was  used  to  assemble  the  tasks  was  to  interview  recon¬ 
naissance  pilots  and  NFO's  who  were  experienced  with  other  similar 
reconnaissance  systems.  The  resulting  task  listings  were  somewhat 
generic,  but  following  the  accumulation  of  experience  with  the  sys¬ 
tems,  the  task  listings  proved  extremely  adequate. 

There  were  several  favorable  conditions  which  existed  in  this 
effort  despite  the  limitations  of  not  having  true  SME '  s .  One  of  these 
conditions  was  that  theSME's  which  were  available  had  diverse  back¬ 
grounds.  There  were  SME's  from  the  Navy's  RA-5  and  RF-8  communities 
as  well  as  from  the  USAF's  RF-4  community.  Additionally,  there  were 
SME's  from  the  F-14  community  available.  This  resulted  In  a  broad 
base  of  knowledge  and  experience  which  was  successfully  exploited. 
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Another  favorable  condition  for  this  particular  task  listing 
was  that  there  was  no  SME  availability  problem.  There  were  reconnais¬ 
sance  pilots  and  NFO's  on  board  in  more  than  sufficient  numbers  to 
meet  the  requirements  of  this  program.  The  ease  with  which  the  tasks 
were  assembled  and  the  resulting  quality  of  the  task  listing  is 
directly  attributable  to  the  ready  availability  of  the  SME ' s .  There 
was  far  less  of  a  problem  in  this  ef  for*  than  in  the  MAS  effort 
during  which  SME  availability  was  extremely  limited.  This  advantage 
continued  throughout  the  development  c  f  the  learning  objectives 
and  the  student  syllabus  and  course  materials  for  the  TARPS  addition. 

In  the  process  of  assembling  t  n e  TARPS  tasks,  the  F-14  (MAS) 
task  listings  were  used  a;,  the  b  -  s  .1  s  for  determining  the  RF-14  tasks. 
The  eight  MAS  mission  phases  wet  -  examined  and  TARPS  peculiar  tasks 
were  added  wherever  needed,  i  ...  v  t  r  ,  c  he  interviews  with  the  recon¬ 
naissance  SME 1 s  yielded  far  more  tasks  than  could  be  sorted  into 
one  of  the  eight  MAS  mission  phases,  the  result  being  the  creation 
of  two  additional  mission  phases  peculiar  to  the  RF-14.  These  mission 
phases  were  titled  MISSION  PLANNING  and  RECONNAISSANCE. 


OBJECTIVES  HIERARCHIES  (TARPS). 

Duration  of  Task:  October,  1978  -  December,  1978 

Manhours  Expended: 


Program  Manager 

2 

Instruct iona 1 

Psychologist  I 

4 

Instructional 

Psychologist  II 

25 

Instruct ional 

T  e  c  h  n  o  1  v ;  .  r-  c  T 

15 

Subject  Matter 

Expert 

0 

Clerical 

30 

TOTAL  . 

.  .  76 

The  TARPS  learning  objective  hierarchies  were  created  through 
the  same  methodology  used  in  creat  lag  the  F-14  MAS  learning  objectives 
hierarchy.  As  in  the  TARPS  task  listing  effort,  no  Subject  Matter 
Experts  experienced  in  the  use  o'  the  TARPS  system  were  available. 
The  learning  objectives  were,  therefore,  based  solely  on  the  task 
listing  and  were  somewhat  biased  by  the  use  of  the  F-8,  RF-4  and 
A-5  SME ' s .  Learning  objectives  were  added  for  the  mission  planning 
phase  and  for  the  reconnaissance  phase.  Due  to  the  experience  en¬ 
countered  in  the  initial  F-14  learning  objectives  development  process, 
the  problem  of  redundant  learning  objectives  which  existed  in  the 
F-14  MAS  objective  hierarchy  was  eliminated.  Again,  as  with  the 
task  listings  development,  the  SML  availability  during  the  learning 
objectives  hierarchy  development  phase  was  excellent  and  contributed 
greatly  to  the  quality  of  the  final  product. 
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TRANSITION  SYLLABUS  (TARPS). 

Duration  of  Task:  September,  1978  -  June,  1979 
Manhours  Expended: 


Program  Manager  8 
Instructional  Psychologist  I  10 
Instructional  Psychologist  II  66 
Instructional  Technologist  I  19 
Subject  Matter  Expert  0 
Clerical  95 


TOTAL . 198 

In  the  planning  stages  of  the  TARPS  program,  provisions  were 
made  for  the  arrival  of  students  in  accordance  with  expected  avail¬ 
ability  of  the  TARPS  equipped  aircraft.  At  that  time,  aircraft  were 
scheduled  to  be  available  prior  to  the  completion  of  the  RF-14  syl¬ 
labus.  For  that  reason  a  separate  task  was  ordered  to  develop  a 
transition  syllabus  which  was  to  be  used  to  train  the  aircrews  who 
arrived  prior  to  the  completion  of  the  fully  developed  syllabus. 

Several  assumptions  and  constraints  were  attached  to  this  effort. 
The  moat  obvious  assumption  was  that  the  transition  syllabus  would 
be  in  use  for  a  short  time  only  and  then  be  replaced  by  the  fully 
developed  syllabi.  This  effort  was  to  be  done  at  minimum  expense 
with  all  training  materials  to  be  of  a  text  or  lecture  medium  with 
minimal  artwork.  The  artwork  which  was  needed  was  produced  by 
the  drafting  department  at  VF-124.  The  contractor  was  to  provide 
instructional  design  guidance,  but  the  bulk  of  the  work  was  to  be 
done  by  the  Navy  SME ' s . 

The  development  of  the  transition  syllabus  Btarted  with  a  review 
of  existing  reconnaissance  training  syllabi;  in  particular  the  Navy 
RA-5  and  USAF  RF-4  syllabi  were  studied.  Using  these  syllabi  and 
the  knowledge  of  the  available  SME '  s ,  objectives  for  the  transition 
syllabus  were  inferred.  Again,  it  should  be  stressed  that  there 
were  no  SME's  with  experience  in  the  RF-14  available. 

After  a  suitable  list  of  objectives  had  been  developed,  they 
were  organized  into  a  hierarchy  based  upon  prerequisite  relation¬ 
ships.  They  were  then  sorted  and  grouped  into  individual  lessons. 
As  stated  previously,  the  media  were  to  be  texts  or  lectures  only. 
Given  that  constraint,  the  process  of  media  selections  was  eliminated. 
Objectives  which  would  have  been  designated  for  other  media  were 
modified  to  be  compatible  with  the  text  and  lecture  format.  Lesson 
specifications  were  developed  and  the  conversion  to  the  training 
texts  and  lecture  guides  followed  immediately. 

Unfortunately,  none  of  these  training  materials  were  ever  used 
by  students.  Due  to  delays  in  the  development  and  testing  of  the 
TARPS  POD  it  became  obvious  that  aircraft  would  not  be  available 
as  originally  scheduled,  so  the  flow  of  reconnaissance  atudents 
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to  VF-124  was  stopped.  The  materials  developed  for  this  transition 
syllabus  were,  however,  used  as  reference  material  during  develop¬ 
ment  of  the  actual  student  syllabus. 


IMPLEMENTATION  SUPPORT. 


Duration  of  Task:  November,  1979  -  June,  1980 

Manhours  Expended : 

Program  Manager  44 

Instructional  Psychologist  T  9 

Instructional  Psychologist  II  58 

Instructional  Technologist  I  28 

Subject  Matter  Expe. ;  34 

Clerical  2  5 

TOTAL . 198 


During  the  periods  of  syllabus  implementation,  support  was  pro¬ 
vided  by  the  contractor  to  both  VF-124  and  at  VF-101.  This  support 
was  required  because  there  were  many  events  in  the  syllabus  which 
had  not  been  properly  validated  during  formative  evaluation  and  be¬ 
cause  the  number  of  instructors  at  both  VF-124  and  VF-101  was  below 
that  required  for  normal  operations.  The  increased  demands  during 
implementation  simply  exceeded  the  capabilities  of  an  already  over¬ 
worked  instructor  corps. 

During  syllabus  development  it  ves  planned  that  each  event  would 
be  formatively  evaluated  and  re  ’too  as  Indicated,  however,  this 
goal  was  not  achieved  due  to  the  lack  of  availability  of  qualified 
students.  As  a  result,  a  great  many  events  needed  to  be  evaluated 
during  the  implementation  of  the  syllabus.  This  necessitated  the 
administration  of  pre-  and  post-tests,  the  interviewing  of  students 
and  the  compilation  and  analysis  of  the  resulting  data.  These  tasks 
were  performed  by  the  contractor. 


98 


NAVTRAEQUIPCEN  78-C-0004-1 


SECTION  IV 
RESULTS 


IMPLEMENTATION 

The  Implementation  of  the  revised  F-14  syllabus  began  at  VP-124 
in  November,  1979.  Prior  to  this  implementation,  the  contractor 
-had  provided  a  final  implementation  plan  which  was  oriented  speci¬ 
fically  toward  VF-124.  The  implementation  of  the  revised  syllabus 
at  VF-101  took  place  in  June,  1980.  A  separate  final  Implementation 
plan  was  produced  for  VF-101.  In  both  cases,  these  implementation 
plans  detailed  the  requirements  in  terms  of  facilities,  manage¬ 
ment,  students  and  instructional  resources.  Further,  the  implemen¬ 
tation  plans  provided  contingency  plans  in  the  event  that  certain 
resources  were  not  available  at  the  required  times. 


PROGRAM  MANAGEMENT 

During  the  performance  of  this  project,  it  was  recognized  by 
the  Navy  that  the  existing  management  system  would  not  be  adequate 
to  meet  the  needs  of  the  new  training  program.  As  a  result,  the 
Navy,  in  conjunction  with  the  contractor,  examined  the  problem  of 
aircrew  training  management  in  detail  and  developed  recommendations 
for  a  reorganization  of  the  existing  operations  department  which 
placed  more  emphasis  on  all  aspects  of  training .  This  reorganization 
has  been  detailed  earlier  in  this  report. 

It  was  also  realized  that  the  ongoing  maintenance  or  quality 
control  of  the  new  aircrew  training  system  was  an  absolute  necessity. 
One  of  the  tasks  involved  in  this  project  was  the  development  of 
specific  quality  control  plans  which  would  ensure  the  continued 
viability  of  the  aircrew  training  program,  once  contractor  support 
was  no  longer  available.  The  Navy  recognized  that  the  optimum  means 
for  maintaining  quality  control  would  be  through  the  use  of  an  in- 
house  educational  specialist  who  could  provide  the  FRS  with  instruc¬ 
tional  technology  support  through  the  outyears. 

One  of  the  tools  which  both  the  squadron  and  the  educational 
specialist  will  use  for  the  program  maintenance  is  the  Automated 
Training  and  Support  System  (ATSS).  This  computerized  data  base 
and  management  system  provides  management,  bookkeeping  and  scheduling 
services  to  the  squadron.  It  was  envisioned  that  each  objective 
in  the  training  syllabus  could  be  entered  into  this  data  base  and 
quickly  accessed  if  the  need  arose  for  revisions  to  an  objective  or 
to  the  training  materials  associated  with  an  objective.  The  ATSS 
was  also  designed  to  maintain  records  of  each  student's  progress 
and,  based  upon  that  progress,  to  develop  and  print  the  dally  flight 
and  training  schedules.  The  method  of  coding  each  of  the  events 
in  the  syllabus  was  designed  specifically  to  allow  entry  of  the  event 
codes  into  the  ATSS. 
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STUDENT  SCHEDULING 

The  events  in  the  revised  F-14  training  syllabus  were  organized 
into  a  syllabus  hierarchy  which  allowed  the  students  to  enter  the 
training  program  at  multiple  points,  all  at  a  fairly  low  level,  and 
to  progress  through  the  syllabus  on  varying  paths.  The  reason  for 
this  hierarchical  syllabus  was  to  alleviate  problems  which  had  been 
discovered  in  the  early  phases  of  resource  analysis.  The  previous 
F-14  syllabus  was  linear  in  nature  in  thn  the  students  would  pro¬ 
gress  from  one  class  to  the  next  without  deviation.  If,  however, 
the  resources  for  the  following  class  were  not  available,  the  students 
were  forced  to  mark  time  in  their  training.  With  the  hierarchical 
syllabus,  the  lack  of  resources  for  a  following  event  does  not 
necessarily  cause  a  stoppage  in  training.  There  are  many  possibilities 
within  the  hierarchical  syllabus  for  alternate  training  events  to 
be  given  to  the  students  if  the  resources  for  the  primary  training 
event  are  not  available. 

The  performance  of  this  function,  as  stated  previously,  was  to 
be  accomplished  using  the  ATSS,  however,  at  the  time  of  implementa¬ 
tion,  the  ATSS  did  not  have  the  capability  to  perform  this  function. 
For  this  reason,  contractor  support  in  the  area  of  student  scheduling, 
was  provided.  An  optimum  sequence  of  events  was  developed  and 
the  first  class  of  students  was  scheduled  in  conformance  with  that 
sequence.  Contractor  support  personnel  and  the  VF-124  schedules 
officer  manually  scheduled  the  hierarchical  syllabus  when  it  became 
apparent  that  the  resources  needed  for  the  optimum  flow  of  events 
would  not  be  available.  This  occurred  frequently  throughout  imple¬ 
mentation,  as  had  been  expected,  so  despite  the  lack  of  ATSS  capa¬ 
bility,  the  function  was  performed  manually  with  great  success. 

A  final  point  concerning  student  scheduling  concerns  those  stu¬ 
dents  who  were  onboard  VF-124  during  the  initial  period  of  implemen¬ 
tation.  According  to  the  old  »*yllabus  and  to  ,tjfc\e  new  syllabus;  a 
class  of  students  would  begin  the  training  syllabus  during  the  first 
week  of  eaC? Ti  month.  This  created  a  situation  in  which  the  newest 
class  vas  progressing  through  a  different  syllabus  than  the  five 
previous  classes.  The  result  of  this  method  of  implementation,  was 
that  two  separate  syllabi,  teaching  similar  material,  existed  for  a 
period  of  five  months  following  the  initial  implementation  of  the 
revised  syllabus.  By  the  end  of  the  fifth  month  of  implementation, 
the  last  students  who  were  prcgr-ssing  through  the  old  syllabus, 
had  graduated  and  the  entire  aircrew  training  program  was  using 
the  revised  syllabus. 
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PROBLEMS  AFPECTING  IMPLEMENTATION 

As  with  program  development,  there  was  also  a  problem  of  Instruc¬ 
tor  availability  during  implementation.  This  was  caused  by  the  con¬ 
sistent  undermanning  of  the  squadron,  as  noted  elsewhere,  and  by 
the  fact  that  two  months  prior  to  implementation  there  were  successive 
-carrier  qualification  detachments  which  required  the  utilization 
of  a  great  percentage  of  the  Instructors  at  VF-124. 

The  first  of  two  major  affects  of  this  lack  of  availability  was 
that  the  Instructor  Under  Training  (IUT)  program  was  not  completely 
accomplished  prior  to  the  implementation  of  the  syllabus.  It  had 
been  planned  that  much  of  this  training  for  the  Instructors  would  be 
done  in  groups,  but  as  a  result  of  the  lack  of  availability,  the 
training  had  to  be  performed  by  the  contractor  in  a  series  of  one 
on  one  encounters  with  the  instructors.  The  second  effect  of  this 
lack  of  availability  was  that  many  of  the  tasks  which  needed  to  be 
performed  at  the  training  squadron  simply  had  to  be  postponed  or 
overlooked.  There  was,  for  instance,  a  lack  of  capablity  to  staff 
the  academic  center  with  an  Instructor  pilot  or  NFO  on  a  full  time 
basis  as  had  been  proposed. 

Another  problem  resulted  from  the  size  of  the  first  class  to 
use  the  revised  training  syllabus.  It  had  been  expected  that  the 
class  size  would  be  approximately  ten;  those  ten  being  five  pilots 
and  five  NFOs.  For  the  first  class,  however,  the  number  of  pilots 
increased  to  a  total  of  twelve.  The  result  of  this  excessive  number 
of  students  was  a  tremendous  demand  on  the  resources  which  were 
available  for  training  since  the  number  of  trainer  sessions,  the 
number  of  flights,  and  the  number  of  self-instructed  media  had  not 
been  designed  to  meet  a  class  of  this  size.  This  placed  a  tremendous 
burden  on  the  scheduling  mechanism  in  the  squadron. 

The  final  problem  which  affected  implementation  concerned  the 
lack  of  availability  of  resources.  The  primary  resource  problem  was 
that,  due  to  maintenance  or  parts  problems,  the  simulators  were  not 
operational  to  the  extent  required  during  the  period  of  implementa¬ 
tion.  The  result  of  this  lack  of  simulator  availability  was  that 
in  many  cases,  an  Increased  emphasis  in  the  academic  area  was  pro¬ 
vided  where  the  simulators  would  not  be  available  for  the  hands-on 
training.  This  meant  that,  for  the  performance  of  some  tasks,  the 
student  would  receive  extra  academic  instruction  and  proceed  directly 
to  the  airplane.  Those  tasks  which  were  deemed  essential  for  safe 
flight,  however,  were  performed  in  the  simulators.  This  again  added 
to  the  burden  of  scheduling  which  was  being  done  manually  by  the 
schedules  officer  with  contractor  support. 

The  second  major  resource  problem  concerned  the  availability 
of  the  training  materials  necessary  for  the  training  syllabus.  During 
the  period  of  training  materials  development ,  there  had  been  repeated 
delays  in  the  production  of  these  materials.  Despite  these  delays, 
it  had  bean  decided  that  November  of  1979  would  be  the  date  of  im¬ 
plementation,  so  plans  were  developed  for  that  implementation  using. 
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in  some  cases,  preproduction  materials.  This  was  especially  pre¬ 
valent  during  the  first  two  months  of  the  new  syllabus.  During  that 
two  aonth  period  a  maximum  effort  was  put  forward  to  complete  all 
the  remaining  training  materials  for  the  other  four  months  of  the 
training  syllabus. 


EVALUATION 

The  formative  evaluation  process  was  used  to  evaluate  and  revise 
the  lesson  materials  before  implementation.  To  this  end,  the  con¬ 
tractor  conducted  small  group  tryouts  of  a  large  number  of  replicable 
lesson  materials  using  four  to  six  FRS  students.  As  mentioned  earlier, 
it  was  not  feasible  to  formatively  evaluate  all  materials  because 
of  time  and  student  availability  constraints.  There  also  were  a  few 
lessons  that  were  tried  ojt  with  review  teams  as  opposed  to  students. 

Prior  to  implementation,  the  total  number  of  the  events  that 
were  formatively  evaluated  was  265.  The  table  below  summarizes  the 
revisions  to  the  student  training  materials  as  a  result  of  formative 
evaluation. 


Table  5:  Summary  of  Revisions  During  Formative  Evaluation 


TOTAL  SEGMENTS  265 

SEGMENTS  REQUIRING  X  REQUIRING 

REVISION  148  REVISION  56X 

Type  of  Revision 

Content* 

Technical 

Format 

by  Medium 

Slide/tapes  (66) 

9 

38 

0 

(Perspective,  Inc.) 

Texts  (33) 

0 

5 

0 

Lectures  (37) 

12 

8 

2 

Trainers  (38) 

8 

0 

12 

Plights  (53) 

0 

0 

0 

Exams  (38) 

0 

18 

36 

*Xnclu4as  typographical  errors,  prasentatlon,  style,  ate. 
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As  mentioned  earlier ,  much  of  the  formative  evaluation  process 
was  concentrated  at  the  lesson  specification  stage.  In  this  stage 
emphasis  was  placed  on  appropriate  objectives,  accurate  generalities, 
and  practice  and  criterion  test  items. 

The  most  intense  formative  evaluations  took  place  with  slide 
.tapes.  There  were  13.5  contractor  man-hours  spent  for  the  average 
two-carousel  program.  This  amount  was  at  least  doubled  by  the  addition 
of  Perspective  and  Navy  personnel.  Consultants  from  Naval  Personnel 
Research  and  Development  Center  (NPRDC)  also  assisted  in  the  formative 
evaluation  process. 

Internal  quality  control  procedures  concentrated  on  measuring 
the  two  most  critical  variables  of  the  ISD  program: 

a.  Student  performance 

b.  Instructional  materials  effectiveness 

The  ability  of  the  students  to  master  the  course  objectives  and  to 
pass  tests  was  the  primary  vehicle  for  the  evaluation  of  the  effec¬ 
tiveness  of  the  training  program.  Consequently,  student  performance 
scores  in  academics,  trainers  and  flight  events  were  collected  and 
analyzed.  In  addition,  student  attitudes  were  assessed  through 
both  Informal  interviews  and  questionnaires. 

The  Instructional  materials  in  the  training  program  included 
all  the  printed  and  audio  visual  instructional  materials  used  in 
direct  support  of  the  training  effort.  In  evaluating  the  efficiency 
of  the  various  media,  instructional  psychologists  and  technologists 
used  the  following  internal  quality  control  data: 

a.  criterion  scores 

b.  student  ratings 

c.  narrative  comments 

d.  Instructor  ratings 

e.  narrative  reports 

The  following  table,  Table  6  compares  the  test  data  collected 
from  the  first  class  using  the  new  syllabus  and  the  last  class  to 
use  the  old  syllabus.  In  the  area  of  academics,  only  tests  that 
were  normative  (SOP  course  rules,  etc.)  as  opposed  to  criterion 
scores,  were  Included.  These  test  scores  can  also  be  evaluated  in 
terms  of  how  close  they  came  to  the  standard  of  3.0  established  by 
COMNAVAIRPAC. 
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In  luaaary ,  students  did  well  on  academic  tests  ss  well  as 
in  trainers  and  flights.  These  acores,  together  with  inputs  from 
instructor  and  student  questionnaires ,  pinpointed  the  areas  in  need 
of  revision  within  the  syllabus.  Most  revision  fell  into  two  main 
categories  as  discussed  earlier:  technical  accuracy  and  typographical 
errors.  The  third  most  common  type  of  revision  was  to  the  instructional 
.strategy.  Table  7,  summarizes  the  types  of  revisions  occurring  as 
a  result  of  Internal  quality  control. 


Table  7. 


Summary  of 


Bevisions 


to 


Lesson  Materials 


Us>w/»wt 

Bats  Ba arcs 
far  Bawialsa 

far  yams  af  ferrliiaa 

Hll  MS  (gMltln) 
Crams  Cseatry 

Masalas  lattsn 

1.0 

OtaBsac  sarvay/ 

iaaerwiaw 

Ta  casfllsaat  41  BA  M0  tast 
aaB  yrsvlBa  las  true  tar  latrnr- 
face  aai  aort  BBTOPS  practice 
Foils  aat. 

MM  *05  (aMUtM) 
m  M  Itcmn 

* 

ItatiM  tttt  rti.lt •/ 

ftt.rrtMt/ae  rtti m 

IttiUii  lntroetor  tKirfici 
far  t  ikUIuIIt  Mrl.  taV- 
Joet  Mtt.r; 

•It  Mtattcai  tast;  tat; 

trtlatr 

Mt  aat cate. >  tan;  lac; 

trainr;  tan 

AIM  ISO  (MaMMtlM) 
AtreWtU  Mi  asm  war  lag 
PracMurae/Cigh  AOA 
l.f  iMWl 

MT  rarlar/m  sarBsr 
Mart 

Casy  lisa  at  ABBA  MO  4  M0. 
fravMs  las  tractor  elaaaraas 
laterfece  for  tachaically  Blf- 
flealt  4  as f sty  of  flight 
aal>ct . 

BOB  010-170  (fatsat 
rsvlaiss)  sorC^  if 
teat  gas  sums 

lttsmaa 

Blayllfy  test  forsat ,  aa4 
ratfaes  Isas'*  af  taot 

mm  oio-ooo 

(MMHtlM) 

ME  rawlaw/acaBaat 
IttirwlMi 

(cast  la ad  •  aarlas  af  13 
tralasr  swats  caaductlag 
specialised  task  train*  lata 
•  ttalaer  avast a)  prowl Bed 
eaootbsr  asg  meats  af  awaats 
as B  sat  a am  cralasr  Bowies 
awallaBllty  aaaatr slats. 

mm.  on  iisiiihI 

■Milt  OMcrittltM 
tat  1.0 

BB 

Front.  Motm  alt  It  oar.  to* 
call at  hackgraaot  aa  M»«il« 
MCfoaMi/uHlUitlii 

wm  ooo  (•pin) 

Onh  mim  iMttk 

wm  005 

«NO  mil*  9mm 

WW  nrlw/staliit 
last  taaalta/ataOaat 
•array 

(NH  0*0  a*llt  lac.  UH  ON 
•at  N))  ta  *|mN  ta.  mwm- 
lllutat  aat  kintrcklaailf 
ralaiat  takjact  watt,  i ae li¬ 
lt,  tm  (tMtar  aMintMtiat 
•  ittaattM  at  «*Jtn  attat. 

•J 
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In  Table  8,  the  test  data  is  summarized  for  the  third  class  at 
VF-124.  This  data  shows  the  test  items  most  frequently  missed,  and 
the  lessons  in  need  of  revision  as  a  result  of  test  failures*  Accor¬ 
ding  to  the  data  in  this  table,  revisions  need  to  be  made  to  INS, 
Flight  Controls,  Pulse  Modes,  and  Flight  Characteristics. 

There  are  no  statistical  data  representing  the  summary  of  all 
student  and  Instructor  attitudes  to  date.  There  are  however,  some 
conclusions  that  can  be  drawn  at  this  time.  Student  attitudes  toward 
the  syllabus  are  generally  positive.  Most  of  the  instructional  ma¬ 
terials,  at  least  802,  were  well  received.  Students  liked  frequent 
practice  and  feedback  in  the  academic  events,  and  were  especially 
laudatory  about  the  Aircraft  System  and  FAM  trainers.  The  flight 
syllabus  was  reputed  to  be  well  organized  and  logically  supported  by 
prerequisite  events,  however,  the  RIO’s  felt  that  INS  was  initially 
very  difficult  for  them  and  the  pilots  said  at  times  that  they  felt 
pressured  by  flying  FAM  hops  while  learning  weapons  system  academics. 

Instructors  had  both  positive  and  negative  responses  to  the 
new  syllabus.  The  most  encouraging  comments  centered  around  state¬ 
ments  like  "The  best  training  available  in  the  FRS  to  date".  They 
also  appreciated  the  standardization  and  logical  flow  of  events. 
On  the  other  hand,  instructors  felt  that  there  was  not  enough  in¬ 
structor  Interface  with  the  students  and  the  instruction  at  times 
seemed  to  spoonfeed  the  students,  and  in  addition,  they  were  concerned 
about  the  fact  NATOPS  was  not  being  read.  Changes  were  made  to  the 
syllabus  in  order  to  ensure  NATOPS  reading  occurred  in  future  classes. 
Overall  student  performance  on  criterion  referenced  tests  was  excel¬ 
lent.  Similarly,  student  performance  in  trainers  and  on  flights 
was  above  COMNAVAIRPAC  standards. 
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Table  8 


.  Test  Items  Most  Frequently  Mlssed-Class  8005/VF-124 


Teat 

Coding 

T«ft  Title 

Pilot/ 

120 

Test  Xteas  Most 
Fraqusatly  Missed 

Blum  Passed 

By  90X* 

UU  MO-1 

*Ml  lyitHM 

loth 

nose 

y«« 

OK  MO-2 

Twl  *7>tU 
Uit|IKlU 

Bath 

D-3 

yes. 

AIK  050 

■ydreullc 

IjttM 

Bath 

none 

y«« 

ASK  060 

Landing  Gear, 
Brakes,  Bock 

Both 

)3,  )♦ 

f 

ASK  070 

Electrical 

Byetsaa 

Both 

).  *.  7,  1* 

y 

ASK  090 

Aircraft 

Lighting; 

ICS/02 

Both 

Lighting:  A— 7,  B-3b,  B— > 

ECS:  B-9 

y 

ASK  090 

night 

Controls 

Both 

t,  7.  »,  l*-k,e,  )0,  17. 

63 

no 

one  no 

■■•tar  Tut 

Both 

C-4,  6 

T«e 

ABBE  120 

ru«ht 

Ckaractarlitlc* 

loth 

A-i;  1-1,  1;  M-*.  7;  M 

no 

uu  i«o 

Cwtm  tula. 

Both 

2* 

y 

MM  1*1 

tor 

Both 

9-e,  6,  c,  d 

yea 

MM  1)0 

Avionics 

Both 

21,  61,  62,  66 

y 

ASBE  160 

vm 

Both 

),  *,  *,  1*.  1).  It,  20, 
2*.  2),  2* 

21. 

no 

r*u  no 

Airtreft  Bys 

Ex  an 

Both 

K 

y 

VBK  010 

Bader  Theory 

Both 

■OM 

y 

me  020 

MCS  Intro 

Both 

11,  29,  33,  38,  63,  66, 

68 

y 

VSK  MO 

False  Modes 

loth 

Sl-d-3;  38-b;  39;  61-d; 
s;  35,  61,  63,  66,  65 

30-a 

•  ao 

nu  oto 

TM  Mu 

Both 

)-».  21,  U.  )»,  *3.  3* 

y 

na  0)0 

HCt  0TS 

110 

6 

y 

n>  om 

Balsa  Doppler 
Medea 

1X0 

3).  *1,  *3,  31 

y 

HI  000 

uo  sci  riui 

110 

•DM 

y 

SMC  070 

mu  sn  hmi 

rust 

)*#  71,  U,  100,  102 

y 
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STATISTICAL  SUMMARY 

The  following  tables  provide  statistical  data  resulting  from 
this  project.  The  first  table  compares  the  hourly  requirements  of 
the  old  versus  the  new  syllabus.  The  second  table  refines  this  data 
into  a  ratio  of  self-instructed  to  instructor  required  events  In  the 
-old  versus  new  syllabus  and  the  third  table  lists  the  total  number 
of  objectives  resulting  from  this  revision  effort.  The  final  table 
summarises  the  events  and  hours  required  for  the  MAS  and  TARPS 
syllabi  now  in  use. 
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Table  10.  Ratio  of  Self-lnatructed  Events  to  Instructor  Required 
Events  — Conparl son  of  Old  Syllabus  vs  New  Syllabus 


Old  Syllabu« 

Haw  Syllabus 

P 

— 

N 

P 

N 

Salf-Instructad 

20.41 

19.21 

52. 5X 

54.  BX 

79. 6X 

S0.8X 

47. 5X 

45. 2X 

Table  11.  Number  of  Objectives  by  Syllabus  Phase 


Aeadealc 

Trainer 

Flight 

— 

Aircraft  Syittaa  (AS) 

205/1B6 

31/12 

— 

Familiarisation  (FA) 

47/49 

17/13 

101/24 

Waapon  Syatam  (WS) 

47/97 

1/30 

— 

Waapona  (Mlaallaa)  (VM) 

28/28 

— 

— 

Intareapt  Theory  (IT) 

34/32 

— 

— 

Basic  bployaant  (BE) 

35/35 

65/72 

50/65 

Waapona  (Guns)  (WC) 

26/20 

— 

36/9 

Basic  ACM  (BA) 

55/55 

— 

29/16 

Advancad  ACM  (AA) 

23/23 

20/20 

84/90 

Ad  vane ad  Eaployaant  (AE) 

27/27 

29/29 

20/20 

cq/pctp/XR  (cq) 

18/18 

20/20 

18/18 

BXCCE  (BC) 

101/101 

_ 

133/133 

Pilot:  1300 

BF0:  1242 

646/671 

183/196 

471/375 
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Table  12.  Summary  of  Events  and  Ho 


F-14 

i 

RF-14 

TARPS 

Pilot 

NFO 

Pilot 

NFO 

152 

156 

24 

24 

35 

39 

3 

3 

43 

43 

0 

0 

75 

46 

19 

19 

82.5 

82.5 

24.5 

24.5 

55.5 

65.5 

2.5 

2.5 

34.0 

33.0 

9.0 

9.0 

1.5 

1.5 

- 

“ 

173.5 

182.5 

36.0 

36.0 

20.5 

22.5 

3.0 

3.0 

47.0 

46.0 

0 

0 

91.6 

56.6 

31.8 

31.8 

Events 

#  Lesson  Specifications 

#  Exams 

#  Trainers 

#  Flights 

Hours 

Lectures 

Slide/tape 

Text 

Video  Tape 
Total  Academic 
Exams 
Trainers 
Flights 
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SECTION  V 

CONCLUSIONS  AND  RECOMMENDATIONS 


This  section  of  the  Training  Program  Work  Report  will  examine 
.each  of  the  tasks  Involved  In  performing  this  aircrew  training  ISD 
project.  The  problems  which  were  encountered  in  performing  each  of 
these  tasks  will  be  detailed  and  solutions  or  recommendations  for 
Improvements  will  be  specified.  Further,  recommendations  for  changes 
and  improvements  to  the  military  specification  or  data  Item  descrip¬ 
tion  (DID's)  will  be  given.  It  is  hoped  that  these  recommendations 
will  provide  guidance  for  future  ISD  contractors  in  the  performance 
of  similar  projects. 


OBJECTIVES  HIERARCHIES 

Two  major  topics  will  be  discussed  here.  The  first  topic  concerns 
the  modification  of  the  existing  learning  objectives  hierarchies 
from  the  format  specified  in  the  data  item  description  and  specifi¬ 
cation  to  the  format  finally  used  in  the  development  of  the  syllabus. 
This  format  is  referred  to  as  a  syllabus  oriented  learning  objective 
hierarchy.  The  second  point  to  be  discussed  will  be  that  of  the 
formatting  and  numbering  of  the  objectives  hierarchy.  It  was  felt 
by  the  contractor  that  improvements  could  be  made  In  the  display  of 
the  learning  objectives  hierarchy  and  in  the  methods  of  numbering 
the  individual  objectives.  It  is  felt  that  these  recommendations 
will  lead  to  a  more  clearly  defined  relationship  between  subordinate 
and  superordinate  objectives. 

In  the  early  stages  of  this  project,  the  learning  objectives 
hierarchies  were  developed  from  a  previously  developed  task  listing. 
This  task  listing  was  developed  by  examining  each  task  performed 
across  eight  separate  mission  phases.  These  mission  phases  encom¬ 
passed  the  tasks  which  would  be  performed  from  before  takeoff  to 
after  landing.  When  the  objectives  hierarchies  were  developed  from 
this  task  listing,  a  redundancy  of  learning  objectives  occurred.  This 
occurred  because  for  each  task  In  the  task  listing  a  corresponding 
objective  was  developed.  As  an  example,  consider  the  task  "Perform 
Communicat  ion  Using  the  UHF  Radio".  This  task  would  be  found  in  each 
of  the  mission  phases  as  it  occurs  routinely  from  prior-to-f light 
through  post-landing.  As  this  task  did  occur  in  all  eight  mission 
phases,  eight  separate  objectives,  one  for  each  mission  phase,  was 
developed  in  the  learning  objectives  hierarchies.  In  fact,  if  the 
aircrew  once  performed  UHF  communications  satisfactorily,  there  would 
not  be  a  need  to  repeat  this  objective  for  each  stage  of  the  flight. 
The  objective  would  be  satisfied  at  its  first  occurrence. 

As  the  syllabus  events  were  being  developed,  objectives  hier¬ 
archies  were  modified  such  that  this  redundancy  was  eliminated. 
Similar  objectives  were  combined  into  single  objectives  and  placed 
at  an  appropriate  position  in  the  syllabus.  Also,  as  each  lesson 
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specification  was  developed,  and  the  content  of  the  training  events 
became  more  evident,  some  new  objectives  were  added  and  some  objec¬ 
tives  were  eliminated.  This  process  occurred  cont 1 nua 1 ly duri ng  the 
development  of  the  lesson  specifications  and  resulted  in  a  vastly 
different  set  of  objectives  from  those  originally  developed  from 
the  task  listings. 

The  point  to  be  made  here  is  that  these  changes  to  the  learning 
objectives  hierarchies  should  be  anticipated.  Unless  all  knowledge 
concerning  the  system  is  known  beforehand,  it  is  invalid  to  assume 
that  the  learning  objectives  hierachles  developed  from  a  task  listing 
will  be  totally  correct.  The  process  of  lesson  specif  ication  develop¬ 
ment  will  consist  of  interactions  with  various  SME ' s  who  will  in¬ 
variably  improve  or  at  least  revise  some  of  the  objectives  which 
have  been  developed. 

The  second  major  recommendation  concerns  the  format  and  num¬ 
bering  of  objectives  in  the  objectives  hierarchies  which  is  specified 
in  the  data  item  description  and  military  specification. 


ALTERNATIVE  OBJECTIVE  CODING  PROCEDURES 

The  contractor  modified  the  object lve  codi ng  procedure  to  clar¬ 
ify  the  relationships  between  subordinate  objectives  having  two  or 
more  superordinate  ob jec t ives .  The  t wo  sample  objective  hierarchies 
shown  on  the  following  page  illustrate  this  modification.  Pigure 
27  shows  objectives  numbered  according  to  conventions  contained  in 
the  DID.  Consider  the  objective  "Conduct  Mission  Planning”.  While 
the  number  of  this  objective  3. 2. 3.1  shows  that  it  is  prerequisite 
to  objective  3.2.3,  it  does  not  adequately  indicate  that  it  is  also 
prerequisite  to  objective  3.2.4. 

An  Improved  procedure  for  indicating  the  hierarchial  relation¬ 
ships  of  objectives  with  two  or  more  superordinate  objective  is 
shown  in  Figure  28.  The  number  3-2-3, 4-1  for  the  objective  "Conduct 
Mission  Planning"  clearly  indicates  that  it  is  prerequisite  to  both 
objectives  3-2-3  and  3-2-4.  Note  that  this  coding  reference  also 
Includes  a  four  character  alpha  prefix,  originally  used  in  the  task 
i nventory . 

The  coding  procedure  adopted  by  the  contractor  offers  two  ad¬ 
vantages  over  that  contained  in  the  DID.  First,  the  alpha  prefix 
is  the  same  as  that  originally  used  in  the  task  listing  so  that  any 
objective  can  easily  be  traced  back  to  its  related  aircrew  task. 
S e c o if d ,  the  modified  objective  code  facilitates  development  of  com¬ 
puterized  procedures  to  accurately  trace  through  those  branches 
of  tHe  hierarchy  where  a  given  objective  is  prerequisite  to  two  or 
more  superordinate  objectives. 
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Figure  27.  Learning  Objective  with  More 

than  One  Superordinate  Objective 
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Figure  28. 


Modified  Numbering  System  for  a  Learning 
Objective  with  More  than  One  Superordinate 
Object i ve 
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CHANGES  TO  ELIMINATE  REDUNDANCY  AfiD  1  N  C  R  r.  A  S  1  CI.AR1TY  OF  THE  DIAGRAMS 

Two  separate  but  related  modi f 'cet  tons  were  made  In  the  display 
of  the  hierarchy  diagrams .  The  first  Involved  eliminating  redundancy 
found  when  a  hierarchy  must  be  continued  to  subsequent  pages*  The 
second  involved  the  addition  of  a  visual  cue  to  lead  the  reader  to 
•  the  pages  on  which  horizontal  extension  <  •'  the  hierarchy  are  found. 

By  modifying  the  procedure  for  r I s •  ’  ■  ng  ilerarchial  relation¬ 

ships  for  those  objectives  r  e  n  1  r  1  ,  further  analysis  on  another 
page,  the  contractor  eliminated  redundancies  in  the  hierarchy  dia¬ 
grams.  Figure  29  illustrates  the  b  i  e  a r c h i a  1  display  convention 
called  for  by  the  DID.  I ne  j  i  >  s  t  p.i?e  of  a  hierarchy  diagram,  ob¬ 
jective  1.1.1  appears  in  a  "hex  box’  indicating  further  analysis 
on  another  page.  On  the  pxgi  Jv'erc-  the  analysis  for  this  objective 
continues,  the  objecti.-  should  b-.  shown  a «,  the  firBt  solid  line 
box  on  that  page.  All  oblecl.ve-s  show.  «s  r  u  r  d  i  na  t  e  to  the  one 

in  the  hex  box,  page  one,  must  • ■  w-  he  sh.wu  a<  broken  line  boxes 
above  the  solid  line  box  on  pago  2. 

Figure  31  illustrates  th  nr  titled  hierarchy  display  format. 
This  format  makes  use  of  the  objective  <  nitng  procedure  described 
above.  Thus,  the  need  to  include  all  suo^rurdlnate  objectives  on 
the  second  page  of  the  hierarchy  diagram  Is  eliminated.  The  hex  box 
from  the  first  page  of  the  hierarchy  is  represented  as  a  broken 
line  box  on  the  second  page,  end  the  functional  relationships  between 
the  objective  in  the  hex  box  aid  those  s  rdinate  to  it  are  re¬ 

vealed  on  page  2  by  the  objective  codf  c  that  objective  in  the 
broken  line  box. 

The  second  change  allows  t  c- r  ai;  -objective  hierarchy  to  be  ex¬ 
tended  horizontally  to  more  t  h  a  one  page.  For  example,  during  the 
pre-launch  mission  phase,  th  e  P 1 ’ c t  mu  s  f  pa r f o r m  2  6  7  related  aircraft 
systems  checks.  Individually,  each  check  Is  subordinate  to  the  ter¬ 
minal  objective,  but  collect! vi  ly,  all  are  on  the  same  level  in  the 
hierarchy.  To  solve  this  ‘or-  .  t  problem,  a  horizontal  arrow  box 
containing  the  page  n umbel  for  the  lateral  continuation  of  the  hier¬ 
archy  was  added  to  the  diagram,  '.'his  change  is  also  shown  in  Figure  31. 
The  contractor  recommends  that  che'-e  changes  be  incorporated  in 
the  appropria te  DID .  While  points  2  c  1  are  concerned  only  with  the 
depiction  of  the  hierarchy  block  on  paper,  the  first  point  will  im¬ 
pact  the  ability  of  the  hierarchy  to  be  maintained  on  an  automated 
system.  Since  automation  appears  to  be  the  trend  of  the  present 
and  future,  it  would  seem  appropriate  to  ensure  maximum  ease  in  this 
area . 
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MEDIA  SELECTION 

A  point  to  be  made  concerning  media  selection  in  this  project 
Is  that  In  cases  where  the  optimum  media  selection  could  not  be 
procured,  revisions  to  the  objectives  were  made  in  order  that  they 
might  be  accomplished  by  the  media  which  were  available.  During  the 
'media  selection  process,  each  objective  was  assigned  a  primary  and 
alternate  medium.  In  many  cases,  the  alternate  medium  was  used  or 
the  objective  was  modified  such  that  the  primary  medium  could  be 
changed  to  one  which  was  available.  The  obvious  conclusion  to  be 
drawn  here  is  that  economic  and  pragmatic  constraints  sometimes  over¬ 
rode  the  instructional  theories  being  used  to  develop  the  syllabus. 


QUALITY  CONTROL 

In  a  project  such  as  this,  there  are  two  major  areas  in  which 
quality  control  is  essential.  The  first  area  is  in  development  of 
the  training  syllabus  by  the  contractor.  The  second  area  is  the  main¬ 
tenance  of  the  delivered  product  by  the  user  organization.  Concerning 
the  first  area,  it  was  the  responsibility  of  the  contractor  to  deve¬ 
lop  and  adhere  to  quality  control  procedures  which  are  acceptable 
to  the  contracting  agency.  This  allowed  the  contracting  agency  to 
ensure  that  quality  products  were  being  developed  from  the  start. 

The  second  essential  area  is  that  of  quality  control  of  the 
revised  syllabus  over  its  life  cycle  by  the  user  organization.  This 
type  of  quality  control  procedure  development  requires  that  the 
contractor  analyze  the  methodologies  of  the  user  organization  and 
develop  procedures  which  will  work  within  that  organization.  In  the 
case  of  this  project,  recommendations  were  made  concerning  a  reor¬ 
ganization  of  the  training  management  system  to  achieve  the  goals 
of  continuing  quail ty  in  the  training  system.  These  recommendations 
were  implemented  and  a  reorganization  of  the  operations  department 
took  place. 

A  final  point  to  be  made  concerning  quality  control  is  that 
there  is  no  distinction  made  between  replicable  and  non-replicable 
events.  The  processes  of  formative  evaluation  and  internal  quality 
control  must  deal  with  both  replicable  and  non-replicable  events. 
The  military  specification  and  DID  should  be  expanded  such  that 
procedures  be  developed  for  both  types  of  events. 


LESSON  SPECIFICATION  DEVELOPMENT 

During  this  project  one  lesson  specification  was  developed  for 
each  academic  event  in  the  syllabus.  The  one  recommendation  which 
could  be  made  concerning  the  development  of  lesson  specifications 
in  future  contracts,  is  that  increased  emphasis  should  be  placed 
on  the  practice,  feedback  and  testing  sections  during  generality 
development.  It  was  often  found  that  the  SME's  had  an  easier  time 
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deciding  upon  the  generality  inforaation  if  they  were  asked  to  deve¬ 
lop  the  practice  and  testing  items  initially  rather  than  after  the 
generalities  had  all  been  developed.  In  almost  every  case,  the  SME ' s 
knew  what  information  needed  to  be  tested.  If  the  generalities  were 
developed  beforehand,  without  considering  the  practice  and  test 
items,  the  later  development  of  the  practice  and  test  items  often 
~resulted  in  changes  to  the  generalities. 

As  stated  in  the  previous  paragraph,  reason  specifications  were 
developed  for  each  academic  evert.  They  were  not  developed  for  train¬ 
er  or  flight  events.  It  was  determined  that  the  information  required 
for  a  lesson  specification  ror  a  trainer  or  flight  event  would  consist 
of  a  plan  of  events  from  which  a  briefing  or  trainer  guide  could 
be  developed.  It  quickly  became  apparent  that  this  plan  of  events 
was  directly  trans f e r r a b 1 e  in,o  the  final  format  of  the  trainer  or 
flight  events.  It  was  deemed  unnecessary  to  produce  a  separate  doc¬ 
ument,  the  lesson  specification,  when  the  identical  information 
would  be  part  of  the  delivered  training  material.  In  the  case  of 
the  trainer  and  flight  briefing  guide,  reference  was  made  back  to 
the  applicable  academic  lesson  specifications  if  that  reference  ma¬ 
terial  was  needed. 


STUDENT  SYLLABI 

This  task  resulted  in  the  development  of  two  separate  but  closely 
related  syllabi.  One  for  the  pilot  and  one  for  the  NFO.  In  both 
cases,  the  events  in  the  syllabi  were  arranged  in  hierarchical  fashion 
based  upon  prerequisite  relationships.  This  combination  of  separate 
but  related  syllabi,  organized  in  a  hierarchical  fashion,  yielded 
two  desired  results.  The  first  result  was  that  the  front  end  loading 
which  was  so  prevalent  in  the  existing  F-14  syllabus  was  reduced. 
The  time  from  start  of  syllabus  until  first  flight  was  maintained 
or  reduced  and  yet  the  academic  requirements  were  not  increased. 
Further,  the  hierarchical  arrangement  of  the  syllabus  provided  in¬ 
creased  flexibility  in  the  scheduling  of  the  events  in  the  syllabi. 
If  the  resources  to  perform  the  desired  event  were  not  available, 
the  syllabus  was  structured  such  that  alternate  events  could  be 
given  for  which  the  resources  were  available. 


IMPLEMENTATION 

One  of  the  primary  problems  associated  with  implementation  was 
that  it  had  to  occur  without  disturbing  the  ongoing  training  program. 
In  the  original  F-14  training  syllabus,  a  student  would  enter  the 
syllabus  and  graduate  six  months  later.  A  new  class  of  students 
was  starting  the  first  week  of  each  month,  therefore,  during  a  six 
month  period  of  time,  six  classes  would  have  begun.  In  implementing 
the  revised  syllabus,  the  problem  of  having  two  different  syllabi 
running  concurrenlty  existed.  The  students  who  had  begun  the  training 
syllabus  prior  to  implementation  were  some  portion  of  the  way  through 
the  old  syllabus.  It  was  decided  early  on  that  it  was  advantageous 
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to  have  the  students  start  from  the  first  day  with  the  revised  syl¬ 
labus  rather  than  to  implement  piecemeal  or  have  them  proceed  part 
way  through  one  syllabus  and  then  switch  over  to  the  revised  syllabus- 
implementation  was,  therefore,  somewhat  complicated  during  the  in¬ 
itial  administration  of  the  course  until  the  last  student  who  had 
started  on  the  old  syllabus  had  graduated. 

The  most  significant  problem  which  occurred  in  coordinating 
these  activities  was  the  inability  to  perform  the  desired  instructor 
training  prior  to  the  implementation  date.  The  demanding  work  sche¬ 
dule  of  the  instructors  at  the  squdron  continuously  interfered  with 
the  efforts  to  provide  this  training.  Implementation  Itself  was 
jeopardized  by  successive  carrier  qualification  detachments  just 
prior  to  the  date  scheduled  for  implementation.  These  detachments 
required  the  absence  from  the  squadron  of  approximately  50X  of  the 
instructors.  These  problems  required  a  day  to  day  adjustment  by 
the  contractor  and  by  the  available  Navy  personnel. 

Concerning  prior  scheduling,  it  was  felt  that  the  initial  im¬ 
plementation  plan  had  been  called  out  too  early  in  the  syllabus 
development  sequence  to  allow  an  accurate  plan  to  be  developed.  It 
was  felt  that  a  conceptual  implementation  plan  would  have  been  more 
timely  following  the  initial  delivery  of  the  TSRA.  After  that,  a 
more  detailed  Implementation  plan  would  have  been  appropriate  fol¬ 
lowing  the  completion  of  approximately  two-thirds  of  the  lesson 
specification  documents.  At  that  time,  the  contractor  had  an  extremely 
accurate  view  of  the  syllabus  and  also  had  become  familiar  with 
the  daily  operation  of  the  squadrons  at  which  the  implementation 
was  to  take  place.  The  final  implementation  report  would  then  have 
consisted  of  an  update  which  would  have  been  published  just  before 
the  actual  occurance  of  the  implementation.  In  this  contract,  two 
separate  but  similar  implementation  plans  were  introduced.  One  plan 
was  for  VF-124  and  one  for  VF-lOi. 


TRAINER  MODIFICATION  AND  UTILIZATION 

The  purpose  of  the  trainer  modification  and  utilization  report 
was  to  specify  how  the  training  devices  were  utilized  and  if  modifi¬ 
cations  could  be  made  to  those  devices  which  would  Increase  their 
utility.  One  major  assumption  was  made  prior  to  the  development 
of  this  report.  That  assumption  was  that  no  major  modifications 
would  be  made  to  the  existing  suite  of  trainer  devices.  Partly  as 
a  result  of  this  assumption,  and  partly  because  of  the  lmmenent  de¬ 
livery  of  the  device  2F112  F-14  WST,  no  recommendations  for  modifi¬ 
cations  either  major  or  minor  were  made.  It  was  anticipated  that 
the  existing  training  devices  in  conjunction  with  anticipated  capa¬ 
bilities  of  the  WST  would  satisfy  all  the  training  requirements 
of  the  F-14  community. 
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STUDENT  TRAINING  COURSE  MATERIALS 

The  student  training  course  materials  were  the  end  product  of 
this  process  of  revision  of  the  F-14  syllabus.  Each  academic  trainer 
and  flight  event  was  provided  with  new  training  materials  which  com¬ 
plied  with  the  standards  required  for  this  project.  One  of  the  major 
changes  which  took  place  concerning  the  training  materials  was  that 
criterion  reference  testing  replaced  the  norm  referenced  testing 
which  was  in  existence.  While  it  was  =  1 1 1 1  a  requirement  within 
the  squadron  that  the  students  b e  ranked  according  to  their  perfor¬ 
mance,  the  basis  for  this  ranging  was  shifted  from  the  academic  en- 
environment  to  the  hands-on  environments  of  trainer  and  flight  events. 
This  ranking  was  achieved  by  rating  the  performance  of  the  students 
after  they  had  reached  criterion. 


INSTRUCTOR  TRAINING  COURSE  MATERIALS 

In  the  case  of  the  instructor  course  materials,  two  separate 
tasks  were  joined  to  produce  one  deliverable  product.  Those  tasks 
concerned  the  instructor  training  course  materials  and  the  training 
device  ins t r uc t or /o pe r a t o r  training  materials.  It  was  considered 
efficient  to  combine  the  materials  becar.sc  the  recipients  for  the 
two  sets  of  material  were  in  fact  the  same  set  of  instructors. 
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SECTION  VI 

RESOURCE  UTILIZATION 


This  section  of  the  report  details  the  resources  which  were  ex¬ 
pended  during  the  course  of  this  project.  The  following  tables 
are  Included: 

Table  13:  Task  Manpower  and  Duration 

Table  14:  Monthly  Listing  of  Mandays  Per  Task 

Table  15:  Chronological  List  of  Delivered  Reports 

The  project  was  performed  primarily  in  San  Diego,  CA  with  much 
of  the  work  being  performed  on-site  at  VF-124  NAS  Miramar  where 
office  spaces  were  provided  for  contractor  personnel.  These  spaces 
were  used  primarily  during  the  lesson  specification  development  phase 
at  which  time  the  majority  of  SME  interaction  took  place. 

The  equipment  which  was  utilized  during  this  project  consisted 
of  normal  office  equipment,  i.e.,  electric  typewriters,  copying  ma¬ 
chines  and  a  Lexitron  Videotype  94  word  processor.  This  machine  con¬ 
sists  of  a  CRT  display,  a  cassette  tape  record-and-playback  unit 
and  a  high  speed  printer  and  was  used  for  the  production  of  all  the 
lesson  specifications.  As  documents  which  were  reviewed  and  revised 
repeatedly,  the  lesson  specifications  were  far  more  efficiently 
handled  this  way  then  they  could  have  been  on  normal  typewriters. 

A  second  word  processing  machine,  a  Compugraphlcs  Editwriter 
7500,  was  used  for  the  production  of  authored  materials  of  the  print 
medium.  This  machine  allowed  the  use  of  a  great  variety  of  type 
styles  and  sizes  as  well  as  a  variety  of  symbols,  lines,  etc. 
Student  texts,  workbooks,  and  briefing  guides  were  produced  using 
this  machine. 
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Table  15.  Chronological  List  of  Delivered  Reports 


|au 

Tltl. 

1  so*  77 

Nadia  lalactlan  1a port 

1  SOV  77 

r-1*  Alrcrav  Behavioral  Objective* 

J  JM  78 

Quality  Control  Flan  (Draft) 

1)  m  78 

ME  TralRlui  Materials 

18  m  78 

Tralilai  Support  Ka^ulraatati  Analysis  (TSAA) 

i«  an  78 

Student  Syllabi 

iin  78 

M  Tr ft lalo«  Material* 

8  m  78 

Tralaar  Modification  and  Utillaatlon  Report 

U  m  78 

Inplaae  station  Plan 

1  jm  78 

Laaaoo  Formatting  and  Madia  Salactlon 

11  ODG  78 

Coamants  on  Military  Specification  and  DID* a 

*  OCI  78 

Quality  Control  Plan,  lot  Update 

17  OCT  78 

TUPS  Problem  Analysis 

11  OCT  78 

Objectives  Rlarnrchlaa  Update  Report 

10  SOT  78 

Syllafcua  Opdaca  H 

IS  JOS  79 

T087S  Ofcjactl.it 

1  H08  79 

■7-1*  TSU 

1  HO*  79 

*7-10  St adapt  Syllabi 

If  KOI  79 

Instructor  Training  Course  Materials  and  Trslnlm 
Device  Xnatruc tor /Opera tor  Training  Materlnla 

1  HOT  79 

Quality  Control  Plan,  2nd  Update 

1  817  79 

Implementation  Plan  (Draft) 

1  SOV  79 

Quality  Control  Plan  (Pinal) 

1  JOl  79 

P-1*  tARPS  Transition  Syllabus  Development 

1  OUC  79 

Instructor  Under  Trslnlm  Manual 

1  OCT  79 

lap laments tine  Plan  (Pinal) 

1  SOV  79 

TBX> 

1  SOV  79 

Object  loon  llersrehie* 

1  SOV  79 

Student  Syllabi,  2nd  Update  .* 

1  OR  80 

97-101  lapltatDiitlon  71on 

1  Jm  so 

Studant  Traill  18  Cauraa  Laaaaa  Hatarlala 

11  JOt  so 

Tralil  18  Trans  Sfctfc  Safari 

JO  one  80 

Quality  Control  laport 

JO  817  80 

Tralnloo  Support  fcaqulraaanta  Analyala  Updata 

126 


NAVTRAEQUIPCEN  78-C-0004-3 


APPENDIX  A 

SME  TRAINING  MATERIALS,  EXAMPLE 


i 


NAVTRAEQUIPCEN  78-C-0004-1 


MODULE  2:  LESSON  1 
THE  LESSON  SPECIFICATION 


OBJECTIVES 


1.  At  the  completion  of  this  module  you  will  be 
able  to  match  all  the  components  of  a  lesson 
specification  with  their  appropriate  definitions. 


BACKGROUND 

There  Is  a  great  deal  of  good,  complete,  well-designed  lesson  material 
existing  In  the  readiness  squadron  at  this  time;  not  only  In  the  form  of  lesson 
guides  and  sound/slide  programs,  but  also  In  the  heads  and  memories  of  the 
squadron  instructors.  It  Is  not  the  desire  of  Veda  that  this  vital  technical 
Information  be  forgotten  In  the  design  of  the  new  Instructional  program.  Ve 
need  this  information.  In  order  for  this  training  program  to  truly  be  an  im¬ 
provement  over  what  is  now  in  existence,  this  existing  material  must  be  collec¬ 
ted,  organized  In  a  fashion  optimal  for  Instruction,  documented  so  that  It 
can  be  located,  used  and  modified  as  necessary. 

The  lesson  specification  Is  the  format  chosen  for  organizing  and  docu¬ 
menting  technical  information  as  it  relates  to  each  specific  lesson  or  event 
within  the  syllabus.  Academic  events,  trainer  events  and  flight  events  each 
require  a  lesson  specification.  Each  specification  Indicates  the  precise  lesson 
content,  resource  requirements  and  practice  and  testing  provisions  for  the 
lesson.  It  Is  not  the  lesson  Itself.  The  student  will  never  sec  It.  It  Is 
for  use  as  a  blue  print  by  the  lesson  author  when  developing  the  actual  lecture 
guide,  sound/slide  program,  briefing  guide,  etc. 

Ideally,  the  lesson  specification  should  be  written  so  that  the  lesson 
author,  knowing  little  or  nothing  about  the  technical  content,  could  still 
write  a  lesson  from  the  material  provided  In  the  specification.  If  you  arc 
getting  a  little  jittery  at  this  point,  skeptlesl  of  how  somsone  with  no  tech¬ 
nical  background  can  write  lessons  for  the  F-14  weapons  system,  relax  for 
awhile;  this  Is  not  really  going  to  happen.  Fortunately,  the  people  writing 
the  lessons  for  this  training  program  have  a  great  deal  of  aircraft  and  weapon 
systems  knowledge;  both  with  the  F-14  specifically,  and  with  other  aircraft 
such  as  the  F-4.  They  know  the  lenguege  and  the  operational  context  In  which 
the  lessons,  the  Instructors  and  the  students  exist,  but  they  do  not  heve 
day-to-day,  first-hand  intimate  knowledge  of  the  F-14.  Therefore,  they  are 
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relying  on  you,  the  subject  Batter  expert,  for  thorough  and  thoughtful  review 
of  lesson  specification  content  to  ensure  that  all  vital,  technical  content  is 
Included  in  each  lesson.  The  result  of  this  process  will  be  that  when  the 
lesson  is  written,  the  content  of  that  lesson  is  already  organized,  making 
the  authoring  process  more  efficient  and  precluding  valuable  technical  infor- 
oation  from  being  forgotten  or  Inadvertantly  omitted.  Each  subject  matter  ex¬ 
pert  will  be  asked  to  provide  inputs  to  the  lesson  specifications  for  the 
subject  areas  where  he  is  most  knowledgeable.  In  other  words,  you  are  Veda's 
expert  consultant;  and  Veda,  as  well  as  future  F-14  students  and  Instructors, 
rely  on  you  for  precise,  detailed  lesson  content. 

The  following  parts  of  this  lesson  describe  the  lesson  specification  in 
order  for  you  to  more  fully  understand  its  contents.  A  sample  lesson  spe¬ 
cification,  developed  for  a  sound  slide  program  on  the  radar  antenna  is  at¬ 
tached  following  this  lesson,  to  provide  an  example  and  be  used  86  a  reference 
when  completing  this  lesson. 


CONTENT 


The  following  illustration  depicts  the  components  of  the  lesson  speci¬ 
fication  and  how  they  relate  to  one  another.  The  subject  matter  expert  will 
be  asked  for  Inputs  only  to  the  instructional  content  and  Instructional  sup¬ 
port  portions. 


THE  LESSON  SPECIFICATION 


X  •/  a  r  ■ 
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The  lesson  spsdflcstion  Is  eoaposed  of  the  following  parts: 

LESSON  DESCRIPTION:  This  material  Is  made  up  of  administrative 
information  which  outlines  the  lesson  end  describes  its  place  In 
the  syllabus.  The  specific  components  of  the  lesson  description 
•re: 


Introduction;  A  brief  statement  of  the  contents  of  the  lesson 
and  how  it  relates  to  those  lessons  preceding  and  subsequent 
to  it. 

Prerequisite  Information  Map:  A  block  diagram  illustrating 
the  aequence  of  lessons  and  tests  immediately  surrounding  the 
lesson  in  question. 

Lesson  Hierarchy:  A  block  diagram  illustrating  the  sequence 
in  which  the  lesson  objectives  will  be  taught. 

Lesson  Segments  and  Evaluation:  A  statement  of  how  objectives 
are  grouped  within  a  lesson  and  where  they  are  tested. 

Lesson  Objectives:  A  listing  of  the  objectives  within  the 
lesson. 

Madia  and  Resource  Worksheet:  A  coded  matrix  enabling  pre¬ 
cise  specification  of  the  resource  needs  for  the  lesson. 


INSTRUCTIONAL  REQUIREMENT:  The  lesson  objectives  specify  the  re¬ 
quirements  for  instruction  within  the  lesson. 

Lesson  Objectives:  Each  objective  contained  within  the  lesson 
description  is  listed  separately  and  prescribes  the  instruc¬ 
tional  content. 


INSTRUCTIONAL  CONTENT:  This  portion  of  the  lesson  specification 
describes  the  information  that  must  be  delivered  to  the  student 
in  order  for  him  to  meet  his  objectives. 

Generality:  A  precise  statement  of  the  instructional  content 
which  must  be  delivered  to  the  student  in  order  for  him  to 
attain  the  objective.  The  generality  takes  different  forms, 
depending  on  the  type  of  learning  required  in  the  objective. 
The  three  types  of  generalities  are  memory,  concept  and  rule 
using. 
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Generality  Support:  An  expanded  statement  of  instructional 
content  which  attempts  to  increase  student  understanding  of 
the  subject  matter.  The  generality  support  corresponds  to  the 
type  of  generality.  A  specific  type  of  generality  support  is 
required  for  each  of  the  three  types  of  generalities. 


INSTRUCTIONAL  SUPPORT:  The  examples,  which  describe  the  lesson  con¬ 
tent  and  practice  and  testing  features  help  to  enhance  student 
learning. 

Instance  Specification :  Provides  the  type  and  format  of  exam¬ 
ples  to  be  presented  to  the  student  in  order  to  enhance  the 
exact  meaning  of  the  instructional  content.  Instance  specifi¬ 
cations  are  required  only  for  concept  or  rule-using  generali¬ 
ties.  They  Include  several  parts: 

1.  Type  description:  describes  the  examples 
required  to  illustrate  the  instructional 
content. 

2.  Format  description:  sets  the  stage  for  the 
exanples,  i.e.,  what  should  the  examples 
look  like? 

3.  Common  error  analysis:  describes  mistakes 
students  often  make  in  comprehending  the 
material. 

4.  Mastery  criterion  set:  describes  the  types 
and  number  of  examples  required  for  student 
mastery. 

5.  Instance  production:  describes  the  type  and 
number  of  examples  to  be  produced  for  example 
purposes,  practice  purposes  and  testing. 

6.  Testing  criteria:  states  the  level  of  per¬ 
formance  required  of  the  student  in  order 
to  prove  he  has  learned  the  material. 

Practice  and  Testing:  Provides  directions  and  sample  ques¬ 
tions  for  use  within  the  lesson  as  practice  and  later  for  more 
formal  testing.  This  page  is  used  for  memorization  generali¬ 
ties  only. 
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You  ore  needed  by  Veda  In  order  to  evaluate  the  generality  for  technical 
accuracy)  remembering  that  the  generality  ie  the  critical  Information  that 
must  be  delivered  to  the  atudent  ao  that  he  can  meet  the  objective.  Then, 
working  with  a  Veda  repreaentatlve,  you  will  be  writing  generality  support 
and  Instance  specifications.  Veda  will  provide  the  mastery  criterion  set, 
Instance  production  and  testing  criteria  portions  of  the  instance  specifica¬ 
tion. 


.  The  following  lessons  in  this  module  more  precisely  describe  the  gener¬ 
ality,  generality  support.  Instance  specification  and  practice  and  testing  re¬ 
quirements  for  the  lesson  specification.  First,  take  a  look  at  the  specifica¬ 
tion  for  the  radar  antenna  and  become  familiar  with  what  It  looks  like.  If 
you  have  any  questions,  ask  one  of  the  Veda  representatives  at  the  Miramar 
site  or  call  Veda  offices  in  San  Diego  at  (714)  291-8768. 
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PRACTICE 


Match  the  lesson  specification  component  to  its  appropriate  definition. 

_  1 .  generality 

_  2.  introduction 

_  3.  instance  specification 

_  4.  generality  support 

_  5.  type  description 

_  6.  common  error  analysis 

_  7.  format  description 

_  8.  practice  and  testing 

_  9.  lesson  hierarchy 

DEFINITIONS 

a.  Describes  the  examples  required  to  illustrate  the  meaning  of 
the  lesson's  instructional  content. 

b.  Describes  oil  stakes  students  often  make. 

c.  A  precise  statement  of  instructional  content. 

d.  The  sequence  in  which  the  lesson  objectives  will  be  taught. 

e.  An  expanded  statement  of  instructional  content. 

f.  Needed  only  for  concept  and  rule-using  generalities. 

g.  Describes  the  format  of  examples. 

h.  Provides  sasple  questions. 

1.  A  general  overview  of  the  lesson 
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